Finite-Sample Analysis of Least-Squares Policy Iteration - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Reports (Technical Report) Year : 2010

Finite-Sample Analysis of Least-Squares Policy Iteration

Alessandro Lazaric
Mohammad Ghavamzadeh
  • Function : Author
  • PersonId : 868946
Rémi Munos
  • Function : Author
  • PersonId : 836863

Abstract

In this paper, we report a performance bound for the widely used least-squares policy iteration (LSPI) algorithm. We first consider the problem of policy evaluation in reinforcement learning, i.e., learning the value function of a fixed policy, using the least-squares temporal-difference (LSTD) learning method, and report finite-sample analysis for this algorithm. To do so, we first derive a bound on the performance of the LSTD solution evaluated at the states generated by the Markov chain and used by the algorithm to learn an estimate of the value function. This result is general in the sense that no assumption is made on the existence of a stationary distribution for the Markov chain. We then derive generalization bounds in the case when the Markov chain possesses a stationary distribution and is $\beta$-mixing. Finally, we analyze how the error at each policy evaluation step is propagated through the iterations of a policy iteration method, and derive a performance bound for the LSPI algorithm.
Fichier principal
Vignette du fichier
lspi-jmlr.pdf (317.34 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

inria-00528596 , version 1 (22-10-2010)

Identifiers

  • HAL Id : inria-00528596 , version 1

Cite

Alessandro Lazaric, Mohammad Ghavamzadeh, Rémi Munos. Finite-Sample Analysis of Least-Squares Policy Iteration. [Technical Report] 2010. ⟨inria-00528596⟩
258 View
104 Download

Share

Gmail Facebook Twitter LinkedIn More