hal-01248675
https://inria.hal.science/hal-01248675
arxiv:1511.05932
[ENS-PARIS] Ecole Normale Supérieure de Paris
[CNRS] CNRS - Centre national de la recherche scientifique
[INRIA] INRIA - Institut National de Recherche en Informatique et en Automatique
[INRIA-ROCQ] INRIA Paris - Rocquencourt
[INSMI] CNRS-INSMI - INstitut des Sciences Mathématiques et de leurs Interactions
[INRIA_TEST] INRIA - Institut National de Recherche en Informatique et en Automatique
[TESTALAIN1] TESTALAIN1
[INRIA2] INRIA 2
[TDS-MACS] Réseau de recherche en Théorie des Systèmes Distribués, Modélisation, Analyse et Contrôle des Systèmes
[PSL] Université Paris sciences et lettres
[INRIA-PSL] INRIA-PSL
[ENS-PSL] École normale supérieure - PSL
[INRIA-ETATSUNIS] Copublications Inria-Etats-Unis
[INRIA-ROYAUMEUNI] INRIA-ROYAUMEUNI
On the Global Linear Convergence of Frank-Wolfe Optimization Variants
Lacoste-Julien, Simon
Jaggi, Martin
ACM : G.: Mathematics of Computing/G.1: NUMERICAL ANALYSIS/G.1.6: Optimization
ACM : I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE/I.2.6: Learning
[MATH.MATH-OC] Mathematics [math]/Optimization and Control [math.OC]
[INFO.INFO-LG] Computer Science [cs]/Machine Learning [cs.LG]
[STAT.ML] Statistics [stat]/Machine Learning [stat.ML]
COMM
The Frank-Wolfe (FW) optimization algorithm has lately re-gained popularity thanks in particular to its ability to nicely handle the structured constraints appearing in machine learning applications. However, its convergence rate is known to be slow (sublinear) when the solution lies at the boundary. A simple less-known fix is to add the possibility to take 'away steps' during optimization, an operation that importantly does not require a feasibility oracle. In this paper, we highlight and clarify several variants of the Frank-Wolfe optimization algorithm that have been successfully applied in practice: away-steps FW, pairwise FW, fully-corrective FW and Wolfe's minimum norm point algorithm, and prove for the first time that they all enjoy global linear convergence, under a weaker condition than strong convexity of the objective. The constant in the convergence rate has an elegant interpretation as the product of the (classical) condition number of the function with a novel geometric quantity that plays the role of a 'condition number' of the constraint set. We provide pointers to where these algorithms have made a difference in practice, in particular with the flow polytope, the marginal polytope and the base polytope for submodular optimization.
2015-12
en
NIPS 2015 - Advances in Neural Information Processing Systems 28
Montreal, Canada