Soft Action Priors: Towards Robust Policy Transfer - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Soft Action Priors: Towards Robust Policy Transfer

Matheus Centa
  • Fonction : Auteur
  • PersonId : 1224332
Philippe Preux

Résumé

Despite success in many challenging problems, reinforcement learning (RL) is still confronted with sample inefficiency, which can be mitigated by introducing prior knowledge to agents. However, many transfer techniques in reinforcement learning make the limiting assumption that the teacher is an expert. In this paper, we use the action prior from the Reinforcement Learning as Inference framework (Levine 2018)-that is, a distribution over actions at each state which resembles a teacher policy, rather than a Bayesian prior-to recover state-of-the-art policy distillation techniques. Then, we propose a class of adaptive methods that can robustly exploit action priors by combining reward shaping and auxiliary regularization losses. In contrast to prior work, we develop algorithms for leveraging suboptimal action priors that may nevertheless impart valuable knowledge-which we call soft action priors. The proposed algorithms adapt by adjusting the strength of teacher feedback according to an estimate of the teacher's usefulness in each state. We perform tabular experiments, which show that the proposed methods achieve state-of-the-art performance, surpassing it when learning from suboptimal priors. Finally, we demonstrate the robustness of the adaptive algorithms in continuous action deep RL problems, in which adaptive algorithms considerably improved stability when compared to existing policy distillation methods.
Fichier principal
Vignette du fichier
aaai23.pdf (5.12 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03976459 , version 1 (07-02-2023)

Licence

Paternité

Identifiants

  • HAL Id : hal-03976459 , version 1

Citer

Matheus Centa, Philippe Preux. Soft Action Priors: Towards Robust Policy Transfer. AAAI 2023 - Thirty-Seventh AAAI Conference on Artificial Intelligence, Feb 2023, Washington DC, United States. ⟨hal-03976459⟩
31 Consultations
27 Téléchargements

Partager

Gmail Facebook X LinkedIn More