Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows


Understanding the geometric properties of gradient descent dynamics is a key ingredient in deciphering the recent success of very large machine learning models. A striking observation is that trained over-parameterized models retain some properties of the optimization initialization. This "implicit bias" is believed to be responsible for some favorable properties of the trained models and could explain their good generalization properties. The purpose of this article is threefold. First, we rigorously expose the definition and basic properties of "conservation laws", which are maximal sets of independent quantities conserved during gradient flows of a given model (e.g. of a ReLU network with a given architecture) with any training data and any loss. Then we explain how to find the exact number of these quantities by performing finite-dimensional algebraic manipulations on the Lie algebra generated by the Jacobian of the model. Finally, we provide algorithms (implemented in SageMath) to: a) compute a family of polynomial laws; b) compute the number of (not necessarily polynomial) conservation laws. We provide showcase examples that we fully work out theoretically. Besides, applying the two algorithms confirms for a number of ReLU network architectures that all known laws are recovered by the algorithm, and that there are no other laws. Such computational tools pave the way to understanding desirable properties of optimization initialization in large machine learning models.
Fichier principal
Vignette du fichier
2307.00144.pdf (380.89 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04150576 , version 1 (04-07-2023)





Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré. Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows. Thirty-seventh Conference on Neural Information Processing Systems, Dec 2023, New Orleans (Louisiane), United States. ⟨hal-04150576⟩
69 View
16 Download



Gmail Facebook X LinkedIn More