An Exact Characterization of the Generalization Error of Machine Learning Algorithms - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

An Exact Characterization of the Generalization Error of Machine Learning Algorithms

Résumé

In this paper, the notion of worst-case datagenerating (WCDG) probability measure, recently introduced in (Zou-AAAI24), is leveraged to obtain an exact expression of the expected generalization gap (or generalization error) for any machine learning algorithm. This exact expression is provided in terms of information measures involving the WCDG probability measure and leads to an upper bound on the generalization error that is equal to the sum of the mutual information and the lautum information between the models and the datasets, up to a constant factor. This upper bound is achieved by a Gibbs algorithm whose parameters satisfy particular conditions. Finally, given a fixed model, it is shown that the empirical risk is a sub-Gaussian random variable when datasets are sampled from the WCDG probability measure. This observation leads to the construction of new generalization guarantees coined (ϵ, δ)-robustness.
Fichier principal
Vignette du fichier
An Exact Characterization of the Generalization Error of Machine Learning Algorithms.pdf (319.26 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04426500 , version 1 (02-02-2024)

Licence

Paternité

Identifiants

  • HAL Id : hal-04426500 , version 1

Citer

Xinying Zou, Samir M. Perlaza, Iñaki Esnaola, Eitan Altman, H. Vincent Poor. An Exact Characterization of the Generalization Error of Machine Learning Algorithms. 2024. ⟨hal-04426500⟩
77 Consultations
106 Téléchargements

Partager

Gmail Facebook X LinkedIn More