Empirical Risk Minimization with Generalized Relative Entropy Regularization - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail (Preprint/Prepublication) Année : 2022

Empirical Risk Minimization with Generalized Relative Entropy Regularization

Résumé

The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a σ-finite measure instead of a probability measure. This assumption leads to a generalization of the ERM-RER (g-ERM-RER) problem that allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. The solution of the gERM -RER problem is shown to be a unique probability measure mutually absolutely continuous with the reference measure and to exhibit a probably-approximately-correct (PAC) guarantee for the ERM problem. For a given dataset, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the gERM -RER problem. Finally, the sensitivity of the expected empirical risk to deviations from the solution of the gERM -RER problem is studied. In particular, the expectation of the absolute value of sensitivity is shown to be upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets.
Fichier principal
Vignette du fichier
ERM-RER-JMLR.pdf (735.92 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03849748 , version 1 (12-11-2022)
hal-03849748 , version 2 (12-06-2023)
hal-03849748 , version 3 (29-11-2023)
hal-03849748 , version 4 (27-02-2024)

Identifiants

  • HAL Id : hal-03849748 , version 1

Citer

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Generalized Relative Entropy Regularization. 2022. ⟨hal-03849748v1⟩
115 Consultations
113 Téléchargements

Partager

Gmail Facebook X LinkedIn More