Analysis of the AutoML Challenge series 2015-2018 - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Book Sections Year : 2018

Analysis of the AutoML Challenge series 2015-2018


The ChaLearn AutoML Challenge 1 (NIPS 2015-ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed by one round of AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This paper analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this paper (data and code) have been made publicly available at
Fichier principal
Vignette du fichier
automl-challenges.pdf (2.21 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-01906197 , version 1 (26-10-2018)


  • HAL Id : hal-01906197 , version 1


Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, et al.. Analysis of the AutoML Challenge series 2015-2018. Frank Hutter; Lars Kotthoff; Joaquin Vanschoren. AutoML: Methods, Systems, Challenges, Springer Verlag, inPress, The Springer Series on Challenges in Machine Learning. ⟨hal-01906197⟩
398 View
1309 Download


Gmail Mastodon Facebook X LinkedIn More