Empirical Risk Minimization with Relative Entropy Regularization - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles IEEE Transactions on Information Theory Year : 2024

Empirical Risk Minimization with Relative Entropy Regularization

Abstract

The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a $\sigma$-finite measure, and not necessarily a probability measure. Under this assumption, which leads to a generalization of the ERM-RER problem allowing a larger degree of flexibility for incorporating prior knowledge, numerous relevant properties are stated. Among these properties, the solution to this problem, if it exists, is shown to be a unique probability measure, mutually absolutely continuous with the reference measure. Such a solution exhibits a probably-approximately-correct guarantee for the ERM problem independently of whether the latter possesses a solution. For a fixed dataset and under a specific condition, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the ERM-RER problem. The generalization capabilities of the solution to the ERM-RER problem (the Gibbs algorithm) are studied via the sensitivity of the expected empirical risk to deviations from such a solution towards alternative probability measures. Finally, an interesting connection between sensitivity, generalization error, and lautum information is established.
Fichier principal
Vignette du fichier
V4-CameraReady.pdf (767.95 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03849748 , version 1 (12-11-2022)
hal-03849748 , version 2 (12-06-2023)
hal-03849748 , version 3 (29-11-2023)
hal-03849748 , version 4 (08-04-2024)

Licence

Attribution

Identifiers

Cite

Samir M Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini. Empirical Risk Minimization with Relative Entropy Regularization. IEEE Transactions on Information Theory, In press, ⟨10.1109/TIT.2024.3365728⟩. ⟨hal-03849748v4⟩

Relations

139 View
146 Download

Altmetric

Share

Gmail Facebook X LinkedIn More