Recursive Least-Squares Learning with Eligibility Traces - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2011

Recursive Least-Squares Learning with Eligibility Traces

Bruno Scherrer
Matthieu Geist

Résumé

In the framework of Markov Decision Processes, we consider the problem of learning a linear approximation of the value function of some fixed policy from one trajectory possibly generated by some other policy. We describe a systematic approach for adapting on-policy learning least squares algorithms of the literature (LSTD, LSPE, FPKF and GPTD/KTD) to off-policy learning with eligibility traces. This leads to two known algorithms, LSTD($\lambda$)/LSPE($\lambda$) and suggests new extensions of FPKF and GPTD/KTD. We describe their recursive implementation, discuss their convergence properties, and illustrate their behavior experimentally. Overall, our study suggests that the state-of-art LSTD($\lambda$) remains the best least-squares algorithm.
Fichier principal
Vignette du fichier
ewrl.pdf (453.58 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00644511 , version 1 (24-11-2011)

Identifiants

  • HAL Id : hal-00644511 , version 1

Citer

Bruno Scherrer, Matthieu Geist. Recursive Least-Squares Learning with Eligibility Traces. European Wrokshop on Reinforcement Learning (EWRL 11), Sep 2011, Athens, Greece. ⟨hal-00644511⟩
159 Consultations
256 Téléchargements

Partager

Gmail Facebook X LinkedIn More