Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimization - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimization

Abstract

The effect of the relative entropy asymmetry is analyzed in the empirical risk minimization with relative entropy regularization (ERM-RER) problem. A novel regularization is introduced, coined Type-II regularization, that allows for solutions to the ERM-RER problem with a support that extends outside the support of the reference measure. The solution to the new ERM-RER Type-II problem is analytically characterized in terms of the Radon-Nikodym derivative of the reference measure with respect to the solution. The analysis of the solution unveils the following properties of relative entropy when it acts as a regularizer in the ERM-RER problem: i) relative entropy forces the support of the Type-II solution to collapse into the support of the reference measure, which introduces a strong inductive bias that dominates the evidence provided by the training data; ii) Type-II regularization is equivalent to classical relative entropy regularization with an appropriate transformation of the empirical risk function. Closed-form expressions of the expected empirical risk as a function of the regularization parameters are provided.
Fichier principal
Vignette du fichier
m86291-daunas final.pdf (296.29 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04097637 , version 1 (15-05-2023)

Licence

Attribution

Identifiers

Cite

Francisco Daunas, Iñaki Esnaola, Samir M Perlaza, H. Vincent Poor. Analysis of the Relative Entropy Asymmetry in the Regularization of Empirical Risk Minimization. (ISIT 2023 - IEEE International Symposium on Information Theory, Jun 2023, Taipei, Taiwan. ⟨10.1109/ISIT54713.2023.10206876⟩. ⟨hal-04097637⟩
91 View
125 Download

Altmetric

Share

Gmail Facebook X LinkedIn More