Lobster: Load Balance-Aware I/O for Distributed DNN Training - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Lobster: Load Balance-Aware I/O for Distributed DNN Training

Abstract

The resource-hungry and time-consuming process of training Deep Neural Networks (DNNs) can be accelerated by optimizing and/or scaling computations on accelerators such as GPUs. However, the loading and pre-processing of training samples then often emerges as a new bottleneck. This data loading process engages a complex pipeline that extends from the sampling of training data on external storage to delivery of those data to GPUs, and that comprises not only expensive I/O operations but also decoding, shuffling, batching, augmentation, and other operations. We propose in this paper a new holistic approach to data loading that addresses three challenges not sufficiently addressed by other methods: I/O load imbalances among the GPUs on a node; rigid resource allocations to data loading and data preprocessing steps, which lead to idle resources and bottlenecks; and limited efficiency of caching strategies based on pre-fetching due to eviction of training samples needed soon at the expense of those needed later. We first present a study of key bottlenecks observed as training samples flow through the data loading and preprocessing pipeline. Then, we describe Lobster, a data loading runtime that uses performance modeling and advanced heuristics to combine flexible thread management with optimized eviction for distributed caching in order to mitigate I/O overheads and load imbalances. Experiments with a range of models and datasets show that the Lobster approach reduces both I/O overheads and end-to-end training times by up to 1.5× compared with stateof-the-art approaches.
Fichier principal
Vignette du fichier
icpp22-81.pdf (900.82 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03718681 , version 1 (09-07-2022)

Identifiers

Cite

Jie Liu, Bogdan Nicolae, Dong Li. Lobster: Load Balance-Aware I/O for Distributed DNN Training. ICPP '22: The 51st International Conference on Parallel Processing, Aug 2022, Bordeaux, France. ⟨10.1145/3545008.3545090⟩. ⟨hal-03718681⟩
211 View
253 Download

Altmetric

Share

Gmail Facebook X LinkedIn More