The Worst-Case Data-Generating Probability Measure in Statistical Learning - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue IEEE Journal on Selected Areas in Information Theory Année : 2024

The Worst-Case Data-Generating Probability Measure in Statistical Learning

Résumé

The worst-case data-generating (WCDG) probability measure is introduced as a tool for characterizing the generalization capabilities of machine learning algorithms. Such a WCDG probability measure is shown to be the unique solution to two different optimization problems: $(a)$ The maximization of the expected loss over the set of probability measures on the datasets whose relative entropy with respect to a \emph{reference measure} is not larger than a given threshold; and $(b)$ The maximization of the expected loss with regularization by relative entropy with respect to the reference measure. Such a reference measure can be interpreted as a prior on the datasets. The WCDG cumulants are finite and bounded in terms of the cumulants of the reference measure. To analyze the concentration of the expected empirical risk induced by the WCDG probability measure, the notion of $(\epsilon, \delta)$-robustness of models is introduced. Closed-form expressions are presented for the sensitivity of the expected loss for a fixed model. These results lead to a novel expression for the generalization error of arbitrary machine learning algorithms. This exact expression is provided in terms of the WCDG probability measure and leads to an upper bound that is equal to the sum of the mutual information and the lautum information between the models and the datasets, up to a constant factor. This upper bound is achieved by a Gibbs algorithm. This finding reveals that an exploration into the generalization error of the Gibbs algorithm facilitates the derivation of overarching insights applicable to any machine learning algorithm.
Fichier principal
Vignette du fichier
JSAIT_camera.pdf (392.81 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04442591 , version 1 (06-02-2024)
hal-04442591 , version 2 (04-04-2024)

Licence

Paternité

Identifiants

Citer

Xinying Zou, Samir M Perlaza, Iñaki Esnaola, Eitan Altman, H. Vincent Poor. The Worst-Case Data-Generating Probability Measure in Statistical Learning. IEEE Journal on Selected Areas in Information Theory, inPress, ⟨10.1109/JSAIT.2024.3383281⟩. ⟨hal-04442591v2⟩
62 Consultations
127 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More