An Exact Characterization of the Generalization Error of Machine Learning Algorithms
Résumé
In this paper, the notion of worst-case datagenerating (WCDG) probability measure, recently introduced in (Zou-AAAI24), is leveraged to obtain an exact expression of the expected generalization gap (or generalization error) for any machine learning algorithm. This exact expression is provided in terms of information measures involving the WCDG probability measure and leads to an upper bound on the generalization error that is equal to the sum of the mutual information and the lautum information between the models and the datasets, up to a constant factor. This upper bound is achieved by a Gibbs algorithm whose parameters satisfy particular conditions. Finally, given a fixed model, it is shown that the empirical risk is a sub-Gaussian random variable when datasets are sampled from the WCDG probability measure. This observation leads to the construction of new generalization guarantees coined (ϵ, δ)-robustness.
Fichier principal
An Exact Characterization of the Generalization Error of Machine Learning Algorithms.pdf (319.26 Ko)
Télécharger le fichier
Origine | Fichiers produits par l'(les) auteur(s) |
---|