Biasing Approximate Dynamic Programming with a Lower Discount Factor - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2008

Biasing Approximate Dynamic Programming with a Lower Discount Factor

Abstract

Most algorithms for solving Markov decision processes rely on a discount factor, which ensures their convergence. It is generally assumed that using an artificially low discount factor will improve the convergence rate, while sacrificing the solution quality. We however demonstrate that using an artificially low discount factor may significantly improve the solution quality, when used in approximate dynamic programming. We propose two explanations of this phenomenon. The first justification follows directly from the standard approximation error bounds: using a lower discount factor may decrease the approximation error bounds. However, we also show that these bounds are loose, thus their decrease does not entirely justify the improved solution quality. We thus propose another justification: when the rewards are received only sporadically (as in the case of Tetris), we can derive tighter bounds, which support a significant improvement in the solution quality with a decreased discount factor.
Fichier principal
Vignette du fichier
finaldiscount.pdf (150.18 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

inria-00337652 , version 1 (07-11-2008)

Identifiers

  • HAL Id : inria-00337652 , version 1

Cite

Marek Petrik, Bruno Scherrer. Biasing Approximate Dynamic Programming with a Lower Discount Factor. Twenty-Second Annual Conference on Neural Information Processing Systems -NIPS 2008, Dec 2008, Vancouver, Canada. ⟨inria-00337652⟩
193 View
496 Download

Share

Gmail Facebook X LinkedIn More