Self-Imitation Advantage Learning - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Self-Imitation Advantage Learning

Johan Ferret
  • Fonction : Auteur
  • PersonId : 1092607
Olivier Pietquin
  • Fonction : Auteur
  • PersonId : 1090627
Matthieu Geist
  • Fonction : Auteur
  • PersonId : 1090629

Résumé

Self-imitation learning is a Reinforcement Learning (RL) method that encourages actions whose returns were higher than expected, which helps in hard exploration and sparse reward problems. It was shown to improve the performance of on-policy actor-critic methods in several discrete control tasks. Nevertheless, applying self-imitation to the mostly action-value based off-policy RL methods is not straightforward. We propose SAIL, a novel generalization of self-imitation learning for off-policy RL, based on a modification of the Bellman optimality operator that we connect to Advantage Learning. Crucially, our method mitigates the problem of stale returns by choosing the most optimistic return estimate between the observed return and the current action-value for self-imitation. We demonstrate the empirical effectiveness of SAIL on the Arcade Learning Environment, with a focus on hard exploration games.
Fichier principal
Vignette du fichier
2012.11989.pdf (3.34 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03159815 , version 1 (04-03-2021)

Identifiants

  • HAL Id : hal-03159815 , version 1

Citer

Johan Ferret, Olivier Pietquin, Matthieu Geist. Self-Imitation Advantage Learning. AAMAS 2021 - 20th International Conference on Autonomous Agents and Multiagent Systems, May 2021, Londres / Virtual, United Kingdom. ⟨hal-03159815⟩
40 Consultations
243 Téléchargements

Partager

Gmail Facebook X LinkedIn More