DataStates: Towards Lightweight Data Models for Deep Learning - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2020

DataStates: Towards Lightweight Data Models for Deep Learning

Bogdan Nicolae

Abstract

A key emerging pattern in deep learning applications is the need to capture intermediate DNN model snapshots and preserve or clone them to explore a large number of alternative training and/or inference paths. However, with increasing model complexity and new training approaches that mix data, model, pipeline and layer-wise parallelism, this pattern is challenging to address in a scalable and efficient manner. To this end, this position paper advocates for rethinking how to represent and manipulate DNN learning models. It relies on a broader notion of data states, a collection of annotated, potentially distributed data sets (tensors in the case of DNN models) that AI applications can capture at key moments during the runtime and revisit/reuse later. Instead explicitly interacting with the storage layer (e.g., write to a file), users can "tag" DNN models at key moments during runtime with metadata that expresses attributes and persistency/movement semantics. A high-performance runtime is the responsible to interpret the metadata and perform the necessary actions in the background, while offering a rich interface to find data states of interest. Using this approach has benefits at several levels: new capabilities, performance portability, high performance and scalability.
Fichier principal
Vignette du fichier
p67_nicolae.pdf (505.83 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

hal-02941295 , version 1 (16-09-2020)

Identifiers

  • HAL Id : hal-02941295 , version 1

Cite

Bogdan Nicolae. DataStates: Towards Lightweight Data Models for Deep Learning. SMC'20: The 2020 Smoky Mountains Computational Sciences and Engineering Conference, Aug 2020, Nashville (virtual conference), United States. ⟨hal-02941295⟩
170 View
452 Download

Share

Gmail Mastodon Facebook X LinkedIn More