Performance Bounds for Lambda Policy Iteration - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Reports (Research Report) Year : 2007

Performance Bounds for Lambda Policy Iteration

Bruno Scherrer

Abstract

We consider the discrete-time infinite-horizon discounted stationary optimal control problem formalized by Markov Decision Processes. We study Lambda Policy Iteration, a family of algorithms parameterized by lambda, originally introduced by Ioffe and Bertsekas. Lambda Policy Iteration generalizes the standard algorithms Value Iteration and Policy Iteration, and has some connections with TD(Lambda) introduced by Sutton & Barto. We consider the discrete-time infinite-horizon discounted stationary optimal control problem formalized by Markov Decision Processes. We study Lambda Policy Iteration, a family of algorithms parameterized by lambda, originally introduced by Ioffe and Bertsekas. Lambda Policy Iteration generalizes the standard algorithms Value Iteration and Policy Iteration, and is closely related to TD(lambda) introduced by Sutton and Barto. We deepen the original theory developped by Ioffe and Bertsekas by providing convergence rate bounds which generalize standard bounds for Value Iteration described for instance by Puterman. We also develop the theory of this algorithm when it is used in an approximate form. Doing so, we extend and unify the separate analyses developped by Munos for Approximate Value Iteration and Approximate Policy Iteration. The main contribution of this paper is that we show that doing Approximate Lambda Policy Iteration is sound.
Fichier principal
Vignette du fichier
lpi.pdf (431.05 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

inria-00185271 , version 1 (05-11-2007)
inria-00185271 , version 2 (09-11-2007)
inria-00185271 , version 3 (13-11-2007)
inria-00185271 , version 4 (03-10-2011)
inria-00185271 , version 5 (11-10-2011)

Identifiers

  • HAL Id : inria-00185271 , version 2
  • ARXIV : 0711.0694

Cite

Bruno Scherrer. Performance Bounds for Lambda Policy Iteration. [Research Report] 2007, pp.35. ⟨inria-00185271v2⟩
331 View
334 Download

Altmetric

Share

Gmail Facebook X LinkedIn More