There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning

Johan Ferret
  • Function : Author
Olivier Pietquin
  • Function : Author
Philippe Preux
  • Function : Author
Matthieu Geist
  • Function : Author
  • PersonId : 790158
  • IdRef : 142341819

Abstract

We propose to learn to distinguish reversible from irreversible actions for better informed decision-making in Reinforcement Learning (RL). From theoretical considerations, we show that approximate reversibility can be learned through a simple surrogate task: ranking randomly sampled trajectory events in chronological order. Intuitively, pairs of events that are always observed in the same order are likely to be separated by an irreversible sequence of actions. Conveniently, learning the temporal order of events can be done in a fully self-supervised way, which we use to estimate the reversibility of actions from experience, without any priors. We propose two different strategies that incorporate reversibility in RL agents, one strategy for exploration (RAE) and one strategy for control (RAC). We demonstrate the potential of reversibility-aware agents in several environments, including the challenging Sokoban game. In synthetic tasks, we show that we can learn control policies that never fail and reduce to zero the side-effects of interactions, even without access to the reward function.
Fichier principal
Vignette du fichier
Reversibility_Aware_Reinforcement_Learning__NeurIPS_.pdf (1.25 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03454640 , version 1 (29-11-2021)

Identifiers

  • HAL Id : hal-03454640 , version 1

Cite

Nathan Grinsztajn, Johan Ferret, Olivier Pietquin, Philippe Preux, Matthieu Geist. There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning. Neural Information Processing Systems (2021), Dec 2021, Virtual, France. ⟨hal-03454640⟩
56 View
67 Download

Share

Gmail Facebook X LinkedIn More