"I'm sorry Dave, I'm afraid I can't do that" Deep Q-Learning From Forbidden Actions - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2019

"I'm sorry Dave, I'm afraid I can't do that" Deep Q-Learning From Forbidden Actions

Résumé

The use of Reinforcement Learning (RL) is still restricted to simulation or to enhance human-operated systems through recommendations. Real-world environments (e.g. industrial robots or power grids) are generally designed with safety constraints in mind implemented in the shape of valid actions masks or contingency controllers. For example, the range of motion and the angles of the motors of a robot can be limited to physical boundaries. Violating constraints thus results in rejected actions or entering in a safe mode driven by an external controller, making RL agents incapable of learning from their mistakes. In this paper, we propose a simple modification of a state-of-the-art deep RL algorithm (DQN), enabling learning from forbidden actions. To do so, the standard Q-learning update is enhanced with an extra safety loss inspired by structured classification. We empirically show that it reduces the number of hit constraints during the learning phase and accelerates convergence to near-optimal policies compared to using standard DQN. Experiments are done on a Visual Grid World Environment and Text-World domain.
Fichier principal
Vignette du fichier
Sorry_Dave__I_m_afraid_I_can_t_do_that.pdf (602.71 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02387419 , version 1 (29-11-2019)
hal-02387419 , version 2 (28-01-2021)

Identifiants

  • HAL Id : hal-02387419 , version 1

Citer

Mathieu Seurin, Philippe Preux, Olivier Pietquin. "I'm sorry Dave, I'm afraid I can't do that" Deep Q-Learning From Forbidden Actions. Workshop on Safety and Robustness in Decision Making (NeurIPS 2019), Dec 2019, Vancouver, Canada. ⟨hal-02387419v1⟩
161 Consultations
491 Téléchargements

Partager

Gmail Facebook X LinkedIn More