Large Scale Caching and Streaming of Training Data for Online Deep Learning - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year :

Large Scale Caching and Streaming of Training Data for Online Deep Learning

Abstract

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.
Fichier principal
Vignette du fichier
Large_Scale_Caching_And_Streaming_of_Training_Data_for_Online_Deep_Learning_CRC.pdf (782.9 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03694669 , version 1 (13-06-2022)

Identifiers

Cite

Jie Liu, Bogdan Nicolae, Dong Li, Justin M Wozniak, Tekin Bicer, et al.. Large Scale Caching and Streaming of Training Data for Online Deep Learning. 12th Workshop on AI and Scientific Computing at Scale using Flexible Computing Infrastructures, Jul 2022, Minneapolis, United States. ⟨10.1145/3526058.3535453⟩. ⟨hal-03694669⟩
70 View
232 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More