A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2012

A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets

Abstract

We propose a new stochastic gradient method for optimizing the sum of a finite set of smooth functions, where the sum is strongly convex. While standard stochastic gradient methods converge at sublinear rates for this problem, the proposed method incorporates a memory of previous gradient values in order to achieve a linear convergence rate. In a machine learning context, numerical experiments indicate that the new algorithm can dramatically outperform standard algorithms, both in terms of optimizing the training error and reducing the test error quickly.
Fichier principal
Vignette du fichier
sag_arxiv.pdf (587.17 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-00674995 , version 1 (28-02-2012)
hal-00674995 , version 2 (05-07-2012)
hal-00674995 , version 3 (06-07-2012)
hal-00674995 , version 4 (11-03-2013)

Identifiers

Cite

Nicolas Le Roux, Mark Schmidt, Francis Bach. A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets. NIPS'12 - 26 th Annual Conference on Neural Information Processing Systems (2012), 2012, Lake Tahoe, United States. ⟨hal-00674995v4⟩
629 View
5086 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More