Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Data - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Data

Abstract

One of the key challenges in decentralized and federated learning is to design algorithms that efficiently deal with highly heterogeneous data distributions across agents. In this paper, we revisit the analysis of the popular Decentralized Stochastic Gradient Descent algorithm (D-SGD) under data heterogeneity. We exhibit the key role played by a new quantity, called neighborhood heterogeneity, on the convergence rate of D-SGD. By coupling the communication topology and the heterogeneity, our analysis sheds light on the poorly understood interplay between these two concepts. We then argue that neighborhood heterogeneity provides a natural criterion to learn data-dependent topologies that reduce (and can even eliminate) the otherwise detrimental effect of data heterogeneity on the convergence time of D-SGD. For the important case of classification with label skew, we formulate the problem of learning such a good topology as a tractable optimization problem that we solve with a Frank-Wolfe algorithm. As illustrated over a set of simulated and real-world experiments, our approach provides a principled way to design a sparse topology that balances the convergence speed and the per-iteration communication costs of D-SGD under data heterogeneity.
Fichier principal
Vignette du fichier
2204.04452.pdf (961.05 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03905091 , version 1 (17-12-2022)
hal-03905091 , version 2 (23-12-2023)

Licence

Attribution

Identifiers

Cite

Batiste Le Bars, Aurélien Bellet, Marc Tommasi, Erick Lavoie, Anne-Marie Kermarrec. Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Data. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics (AISTATS 2023), 2023, Valencia, Spain, Spain. ⟨hal-03905091v2⟩
97 View
120 Download

Altmetric

Share

Gmail Facebook X LinkedIn More