A lower bound and a near-optimal algorithm for bilevel empirical risk minimization - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2024

A lower bound and a near-optimal algorithm for bilevel empirical risk minimization

Résumé

Bilevel optimization problems, which are problems where two optimization problems are nested, have more and more applications in machine learning. In many practical cases, the upper and the lower objectives correspond to empirical risk minimization problems and therefore have a sum structure. In this context, we propose a bilevel extension of the celebrated SARAH algorithm. We demonstrate that the algorithm requires $\mathcal{O}((n+m)^{\frac12}\varepsilon^{-1})$ gradient computations to achieve $\varepsilon$-stationarity with $n+m$ the total number of samples, which improves over all previous bilevel algorithms. Moreover, we provide a lower bound on the number of oracle calls required to get an approximate stationary point of the objective function of the bilevel problem. This lower bound is attained by our algorithm, which is therefore optimal in terms of sample complexity.
Fichier principal
Vignette du fichier
56.pdf (1.39 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04302861 , version 1 (23-11-2023)
hal-04302861 , version 2 (19-02-2024)
hal-04302861 , version 3 (19-02-2024)

Licence

Identifiants

Citer

Mathieu Dagréou, Thomas Moreau, Samuel Vaiter, Pierre Ablin. A lower bound and a near-optimal algorithm for bilevel empirical risk minimization. International Conference on Artificial Intelligence and Statistics (AISTATS), May 2024, Valencia, Spain. ⟨10.48550/arXiv.2302.08766⟩. ⟨hal-04302861v3⟩
196 Consultations
103 Téléchargements

Altmetric

Partager

More