An Exact Characterization of the Generalization Error of Machine Learning Algorithms
Résumé
In this paper, exact expressions for the generalization error of general machine learning algorithms are presented in terms of information measures. These expressions can be broadly divided into two classes. The first one involves a worst-case data generating (WCDG) probability measure, while the second one involves a Gibbs algorithm. These expressions are formed by the sum of three terms. The first term is either a mutual or lautum information induced by the algorithm between the models and the datasets. The second and third terms are different for the two classes. In the first class, the second term compares via Kullback-Leibler (KL) divergence the posterior probability distribution induced by the algorithm on the datasets with a WCDG probability measure; and the third term compares via KL divergence the marginal of such posterior with the same WCDG probability measure. In the second class, the second term compares via KL divergence the distributions induced on the models by the algorithm and a Gibbs algorithm when both have been training with the same dataset; the third term compares via KL divergence the marginal of the distribution on the models induced by the algorithm with the same Gibbs algorithm. For Gibbs algorithms, the second and third terms jointly become either mutual or lautum information, recovering existing results.
Origine | Fichiers produits par l'(les) auteur(s) |
---|