On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2012

On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes

Bruno Scherrer
Boris Lesner
  • Function : Author
  • PersonId : 933391

Abstract

We consider infinite-horizon stationary $\gamma$-discounted Markov Decision Processes, for which it is known that there exists a stationary optimal policy. Using Value and Policy Iteration with some error $\epsilon$ at each iteration, it is well-known that one can compute stationary policies that are $\frac{2\gamma}{(1-\gamma)^2}\epsilon$-optimal. After arguing that this guarantee is tight, we develop variations of Value and Policy Iteration for computing non-stationary policies that can be up to $\frac{2\gamma}{1-\gamma}\epsilon$-optimal, which constitutes a significant improvement in the usual situation when $\gamma$ is close to $1$. Surprisingly, this shows that the problem of ''computing near-optimal non-stationary policies'' is much simpler than that of ''computing near-optimal stationary policies''.
Fichier principal
Vignette du fichier
nips2012.pdf (131.77 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-00758809 , version 1 (29-11-2012)

Identifiers

Cite

Bruno Scherrer, Boris Lesner. On the Use of Non-Stationary Policies for Stationary Infinite-Horizon Markov Decision Processes. NIPS 2012 - Neural Information Processing Systems, Dec 2012, South Lake Tahoe, United States. ⟨hal-00758809⟩
194 View
559 Download

Altmetric

Share

Gmail Facebook X LinkedIn More