Event-Based Control for Online Training of Neural Networks - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue IEEE Control Systems Letters Année : 2020

Event-Based Control for Online Training of Neural Networks

Résumé

Convolutional Neural Network (CNN) has become the most used method for image classification tasks. During its training the learning rate and the gradient are two key factors to tune for influencing the convergence speed of the model. Usual learning rate strategies are time-based i.e. monotonous decay over time. Recent state-of-the-art techniques focus on adaptive gradient algorithms i.e. Adam and its versions. In this paper we consider an online learning scenario and we propose two Event-Based control loops to adjust the learning rate of a classical algorithm E (Exponential)/PD (Proportional Derivative)-Control. The first Event-Based control loop will be implemented to prevent sudden drop of the learning rate when the model is approaching the optimum. The second Event- Based control loop will decide, based on the learning speed, when to switch to the next data batch. Experimental evaluation is provided using two state-of-the-art machine learning image datasets (CIFAR-10 and CIFAR-100). Results show the Event- Based E/PD is better than the original algorithm (higher final accuracy, lower final loss value), and the Double-Event-Based E/PD can accelerate the training process, save up to 67% training time compared to state-of-the-art algorithms and even result in better performance.
Fichier principal
Vignette du fichier
ZHAO-LCSS.pdf (808.99 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02509604 , version 1 (02-03-2021)

Identifiants

Citer

Zilong Zhao, Sophie Cerf, Bogdan Robu, Nicolas Marchand. Event-Based Control for Online Training of Neural Networks. IEEE Control Systems Letters, 2020, 4 (3), pp.773 - 778. ⟨10.1109/LCSYS.2020.2981984⟩. ⟨hal-02509604⟩
228 Consultations
105 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More