Filtering participants improves generalization in competitions and benchmarks - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Filtering participants improves generalization in competitions and benchmarks

Abstract

We address the problem of selecting a winning algorithm in a challenge or benchmark. While evaluations of algorithms carried out by third party organizers eliminate the inventor-evaluator bias, little attention has been paid to the risk of over-fitting the winner's selection by the organizers. In this paper, we carry out an empirical evaluation using the results of several challenges and benchmarks, evidencing this phenomenon. We show that a heuristic commonly used by organizers consisting of pre-filtering participants using a trial run, reduces over-fitting. We formalize this method and derive a semi-empirical formula to determine the optimal number of top k participants to retain from the trial run.
Fichier principal
Vignette du fichier
ESANN__Judging_Competitions__a_Meta_Learning_Perspective.pdf (511.85 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03869648 , version 1 (24-11-2022)

Identifiers

  • HAL Id : hal-03869648 , version 1

Cite

Adrien Pavao, Zhengying Liu, Isabelle Guyon. Filtering participants improves generalization in competitions and benchmarks. ESANN 2022 - European Symposium on Artificial Neural Networks, Oct 2022, Bruges, Belgium. ⟨hal-03869648⟩
57 View
66 Download

Share

Gmail Mastodon Facebook X LinkedIn More