A note on regularised NTK dynamics with an application to PAC-Bayesian training - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

A note on regularised NTK dynamics with an application to PAC-Bayesian training

Résumé

We establish explicit dynamics for neural networks whose training objective has a regularising term that constrains the parameters to remain close to their initial value. This keeps the network in a lazy training regime, where the dynamics can be linearised around the initialisation. The standard neural tangent kernel (NTK) governs the evolution during the training in the infinite-width limit, although the regularisation yields an additional term appears in the differential equation describing the dynamics. This setting provides an appropriate framework to study the evolution of wide networks trained to optimise generalisation objectives such as PAC-Bayes bounds, and hence potentially contribute to a deeper theoretical understanding of such networks.
Fichier principal
Vignette du fichier
2312.13259.pdf (277.66 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY NC SA - Paternité - Pas d'utilisation commerciale - Partage selon les Conditions Initiales

Dates et versions

hal-04358260 , version 1 (21-12-2023)

Licence

Paternité

Identifiants

Citer

Eugenio Clerico, Benjamin Guedj. A note on regularised NTK dynamics with an application to PAC-Bayesian training. 2023. ⟨hal-04358260⟩
19 Consultations
7 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More