Rate of Convergence and Error Bounds for LSTD($\lambda$) - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Reports (Research Report) Year : 2014

Rate of Convergence and Error Bounds for LSTD($\lambda$)

Manel Tagorti
  • Function : Author
  • PersonId : 948730
Bruno Scherrer

Abstract

We consider LSTD($\lambda$), the least-squares temporal-difference algorithm with eligibility traces algorithm proposed by Boyan (2002). It computes a linear approximation of the value function of a fixed policy in a large Markov Decision Process. Under a $\beta$-mixing assumption, we derive, for any value of $\lambda \in (0,1)$, a high-probability estimate of the rate of convergence of this algorithm to its limit. We deduce a high-probability bound on the error of this algorithm, that extends (and slightly improves) that derived by Lazaric et al. (2012) in the specific case where $\lambda=0$. In particular, our analysis sheds some light on the choice of $\lambda$ with respect to the quality of the chosen linear space and the number of samples, that complies with simulations.
Fichier principal
Vignette du fichier
report.pdf (455.47 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-00990525 , version 1 (13-05-2014)

Identifiers

Cite

Manel Tagorti, Bruno Scherrer. Rate of Convergence and Error Bounds for LSTD($\lambda$). [Research Report] 2014. ⟨hal-00990525⟩
262 View
386 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More