A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2007

A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD

Manuel Loth
  • Fonction : Auteur
  • PersonId : 836853
Philippe Preux

Résumé

This paper addresses the issue of policy evaluation in Markov Decision Processes, using linear function approximation. It provides a unified view of algorithms such as TD(lambda), LSTD(lambda), iLSTD, residual-gradient TD. It is asserted that they all consist in minimizing a gradient function and differ by the form of this function and their means of minimizing it. Two new schemes are introduced in that framework: Full-gradient TD which uses a generalization of the principle introduced in iLSTD, and EGD TD, which reduces the gradient by successive equi-gradient descents. These three algorithms form a new intermediate family with the interesting property of making much better use of the samples than TD while keeping a gradient descent scheme, which is useful for complexity issues and optimistic policy iteration.
Fichier principal
Vignette du fichier
unified.pdf (136.29 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

inria-00116936 , version 1 (28-11-2006)
inria-00116936 , version 2 (29-11-2006)

Identifiants

Citer

Manuel Loth, Philippe Preux. A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD. European Symposium on Artificial Neural Networks, Apr 2007, Belgium. ⟨inria-00116936v1⟩
261 Consultations
342 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More