Towards Low-Overhead Resilience for Data Parallel Deep Learning - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Towards Low-Overhead Resilience for Data Parallel Deep Learning

Bogdan Nicolae
Tanner Hobson
  • Function : Author
  • PersonId : 1096317
Orcun Yildiz
  • Function : Author
  • PersonId : 1029719
Tom Peterka
  • Function : Author
  • PersonId : 1096319

Abstract

Data parallel techniques have been widely adopted both in academia and industry as a tool to enable scalable training of deep learning models. At scale, DL training jobs can fail due to software or hardware bugs, may need to be preempted or terminated due to unexpected events, or may perform suboptimally because they were misconfigured. Under such circumstances, there is a need to recover and/or reconfigure data-parallel DL training jobs on-the-fly, while minimizing the impact on the accuracy of the DNN model and the runtime overhead. In this regard, state-of-art techniques adopted by the HPC community mostly rely on checkpoint-restart, which inevitably leads to loss of progress, thus increasing the runtime overhead. In this paper we explore alternative techniques that exploit the properties of modern deep learning frameworks (overlapping of gradient averaging and weight updates with local gradient computations through pipeline parallelism) to reduce the overhead of resilience/elasticity. To this end we introduce a failure simulation framework and two resilience strategies (immediate mini-batch rollback and lossy forward recovery), which we study compared with checkpoint-restart approaches in a variety of settings in order to understand the trade-offs between the accuracy loss of the DNN model and the runtime overhead.
Fichier principal
Vignette du fichier
Towards Low-Overhead Resilience for Data Parallel Deep Learning.pdf (415.05 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03631882 , version 1 (06-04-2022)

Identifiers

  • HAL Id : hal-03631882 , version 1

Cite

Bogdan Nicolae, Tanner Hobson, Orcun Yildiz, Tom Peterka, Dmitry Morozov. Towards Low-Overhead Resilience for Data Parallel Deep Learning. CCGrid'22: The 22nd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, May 2022, Messina, Italy. ⟨hal-03631882⟩
93 View
156 Download

Share

Gmail Facebook X LinkedIn More