Distributed stochastic optimization with large delays - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Mathematics of Operations Research Year : 2022

Distributed stochastic optimization with large delays

Zhengyuan Zhou
  • Function : Author
  • PersonId : 1110071
Nicholas Bambos
  • Function : Author
  • PersonId : 1122330
Yinyu Ye
  • Function : Author
  • PersonId : 1110072


One of the most widely used methods for solving large-scale stochastic optimization problems is distributed asynchronous stochastic gradient descent (DASGD), a family of algorithms that result from parallelizing stochastic gradient descent on distributed computing architectures (possibly) asychronously. However, a key obstacle in the efficient implementation of DASGD is the issue of delays: when a computing node contributes a gradient update, the global model parameter may have already been updated by other nodes several times over, thereby rendering this gradient information stale. These delays can quickly add up if the computational throughput of a node is saturated, so the convergence of DASGD may be compromised in the presence of large delays. Our first contribution is that, by carefully tuning the algorithm's step-size, convergence to the critical set is still achieved in mean square, even if the delays grow unbounded at a polynomial rate. We also establish finer results in a broad class of structured optimization problems (called variationally coherent), where we show that DASGD converges to a global optimum with probability 1 under the same delay assumptions. Together, these results contribute to the broad landscape of large-scale non-convex stochastic optimization by offering state-of-the-art theoretical guarantees and providing insights for algorithm design.
Fichier principal
Vignette du fichier
Main.pdf (6.27 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03342384 , version 1 (13-09-2021)



Zhengyuan Zhou, Panayotis Mertikopoulos, Nicholas Bambos, Peter W Glynn, Yinyu Ye. Distributed stochastic optimization with large delays. Mathematics of Operations Research, 2022, 47 (3), pp.2082-2111. ⟨10.1287/moor.2021.1200⟩. ⟨hal-03342384⟩
127 View
95 Download



Gmail Mastodon Facebook X LinkedIn More