Off-Policy Actor-Critic - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2012

Off-Policy Actor-Critic

Thomas Degris
  • Function : Author
  • PersonId : 934007
Martha White
  • Function : Author
Richard S. Sutton
  • Function : Author

Abstract

This paper presents the first actor-critic algorithm for off-policy reinforcement learning. Our algorithm is online and incremental, and its per-time-step complexity scales linearly with the number of learned weights. Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning. Off-policy techniques, such as Greedy-GQ, enable a target policy to be learned while following and obtaining data from another (behavior) policy. For many problems, however, actor-critic methods are more practical than action value methods (like Greedy-GQ) because they explicitly represent the policy; consequently, the policy can be stochastic and utilize a large action space. In this paper, we illustrate how to practically combine the generality and learning potential of off-policy learning with the flexibility in action selection given by actor-critic methods. We derive an incremental, linear time and space complexity algorithm that includes eligibility traces, prove convergence under assumptions similar to previous off-policy algorithms, and empirically show better or comparable performance to existing algorithms on standard reinforcement-learning benchmark problems.
Fichier principal
Vignette du fichier
Degris_Offpac.pdf (1.46 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-00764021 , version 1 (12-12-2012)

Identifiers

  • HAL Id : hal-00764021 , version 1

Cite

Thomas Degris, Martha White, Richard S. Sutton. Off-Policy Actor-Critic. International Conference on Machine Learning, Jun 2012, Edinburgh, United Kingdom. ⟨hal-00764021⟩
315 View
159 Download

Share

Gmail Mastodon Facebook X LinkedIn More