Estimate Sequences for Variance-Reduced Stochastic Composite Optimization - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2019

Estimate Sequences for Variance-Reduced Stochastic Composite Optimization

Résumé

In this paper, we propose a unified view of gradient-based algorithms for stochastic convex composite optimization by extending the concept of estimate sequence introduced by Nesterov. This point of view covers the stochastic gradient descent method, variants of the approaches SAGA, SVRG, and has several advantages: (i) we provide a generic proof of convergence for the aforementioned methods; (ii) we show that this SVRG variant is adaptive to strong convexity; (iii) we naturally obtain new algorithms with the same guarantees; (iv) we derive generic strategies to make these algorithms robust to stochastic noise, which is useful when data is corrupted by small random perturbations. Finally, we show that this viewpoint is useful to obtain new accelerated algorithms in the sense of Nesterov.
Fichier principal
Vignette du fichier
main.pdf (475.04 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02121913 , version 1 (06-05-2019)

Identifiants

Citer

Andrei Kulunchakov, Julien Mairal. Estimate Sequences for Variance-Reduced Stochastic Composite Optimization. ICML 2019 - 36th International Conference on Machine Learning, Jun 2019, Long Beach, United States. pp.3541-3550. ⟨hal-02121913⟩
133 Consultations
109 Téléchargements

Altmetric

Partager

More