MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization

2021/06/06 by Laurent Condat, Peter Richtárik, Condat, Laurent +1 · 2 citations
Computer Science · Engineering · #Stochastic Gradient Optimization Techniques #Sparse and Compressive Sensing Techniques #Privacy-Preserving Technologies in Data

paper · pdf · doi:10.48550/arxiv.2106.03056

Abstract

We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework, which also makes it possible to design new methods as special cases.

Cited by

• Gradient Projection onto Historical Descent Directions for Communication-Efficient Federated Learning
• The Stochastic Multi-Proximal Method for Nonsmooth Optimization

Related

• A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates
• Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity
• Variance-Reduced Decentralized Stochastic Optimization with Gradient Tracking--Part I: GT-SAGA
• Variance Regularization for Accelerating Stochastic Optimization
• Stochastic Optimization with Random Search