Goal-oriented sensitivity analysis of hyperparameters in deep learning - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue Journal of Scientific Computing Année : 2023

Goal-oriented sensitivity analysis of hyperparameters in deep learning

Résumé

Tackling new machine learning problems with neural networks always means optimizing numerous hyperparameters that define their structure and strongly impact their performances. In this work, we study the use of goal-oriented sensitivity analysis, based on the Hilbert-Schmidt Independence Criterion (HSIC), for hyperparameter analysis and optimization. Hyperparameters live in spaces that are often complex and awkward. They can be of different natures (categorical, discrete, boolean, continuous), interact, and have inter-dependencies. All this makes it non-trivial to perform classical sensitivity analysis. We alleviate these difficulties to obtain a robust analysis index that is able to quantify hyperparameters’ relative impact on a neural network’s final error. This valuable tool allows us to better understand hyperparameters and to make hyperparameter optimization more interpretable. We illustrate the benefits of this knowledge in the context of hyperparameter optimization and derive an HSIC-based optimization algorithm that we apply on MNIST and Cifar, classical machine learning data sets, but also on the approximation of Runge function and Bateman equations solution, of interest for scientific machine learning. This method yields competitive and cost-effective neural networks.
Fichier principal
Vignette du fichier
main.pdf (1.72 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03128298 , version 1 (02-02-2021)
hal-03128298 , version 2 (25-02-2021)
hal-03128298 , version 3 (13-04-2021)
hal-03128298 , version 4 (22-06-2021)
hal-03128298 , version 5 (13-12-2021)

Identifiants

Citer

Paul Novello, Gaël Poëtte, David Lugato, Pietro Marco Congedo. Goal-oriented sensitivity analysis of hyperparameters in deep learning. Journal of Scientific Computing, 2023, 94 (3), pp.45. ⟨10.1007/s10915-022-02083-4⟩. ⟨hal-03128298v5⟩
633 Consultations
842 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More