Efficient Combination of Rematerialization and Offloading for Training DNNs - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

Efficient Combination of Rematerialization and Offloading for Training DNNs

Abstract

Rematerialization and offloading are two well known strategies to save memory during the training phase of deep neural networks, allowing data scientists to consider larger models, batch sizes or higher resolution data. Rematerialization trades memory for computation time, whereas Offloading trades memory for data movements. As these two resources are independent, it is appealing to consider the simultaneous combination of both strategies to save even more memory. We precisely model the costs and constraints corresponding to Deep Learning frameworks such as PyTorch or Tensorflow, we propose optimal algorithms to find a valid sequence of memory-constrained operations and finally, we evaluate the performance of proposed algorithms on realistic networks and computation platforms. Our experiments show that the possibility to offload can remove one third of the overhead of rematerialization, and that together they can reduce the memory used for activations by a factor 4 to 6, with an overhead below 20%.
Fichier principal
Vignette du fichier
offchkpt.pdf (563.2 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03359793 , version 1 (30-09-2021)

Identifiers

  • HAL Id : hal-03359793 , version 1

Cite

Olivier Beaumont, Lionel Eyraud-Dubois, Alena Shilova. Efficient Combination of Rematerialization and Offloading for Training DNNs. NeurIPS 2021 - Thirty-fifth Conference on Neural Information Processing Systems, Dec 2021, Virtual-only Conference, France. ⟨hal-03359793⟩

Collections

CNRS INRIA INRIA2
164 View
264 Download

Share

Gmail Mastodon Facebook X LinkedIn More