Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2020

Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization

Abstract

We consider the setting of distributed empirical risk minimization where multiple machines compute the gradients in parallel and a centralized server updates the model parameters. In order to reduce the number of communications required to reach a given accuracy, we propose a precon-ditioned accelerated gradient method where the preconditioning is done by solving a local optimization problem over a subsampled dataset at the server. The convergence rate of the method depends on the square root of the relative condition number between the global and local loss functions. We estimate the relative condition number for linear prediction models by studying uniform concentration of the Hessians over a bounded domain , which allows us to derive improved convergence rates for existing preconditioned gradient methods and our accelerated method. Experiments on real-world datasets illustrate the benefits of acceleration in the ill-conditioned regime.
Fichier principal
Vignette du fichier
MSR_visit.pdf (3.31 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-02974232 , version 1 (21-10-2020)

Identifiers

  • HAL Id : hal-02974232 , version 1

Cite

Hadrien Hendrikx, Lin Xiao, Sébastien Bubeck, Francis Bach, Laurent Massoulié. Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization. ICML 2020 - Thirty-seventh International Conference on Machine Learning, Jul 2020, Vienna / Virtual, Austria. ⟨hal-02974232⟩
1518 View
119 Download

Share

Gmail Mastodon Facebook X LinkedIn More