Fundamental Design Principles for Reinforcement Learning Algorithms - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Book Sections Year : 2021

Fundamental Design Principles for Reinforcement Learning Algorithms

Abstract

Along with the sharp increase in visibility of the field, the rate at which new reinforcement learning algorithms are being proposed is at a new peak. While the surge in activity is creating excitement and opportunities, there is a gap in understanding of two basic principles that these algorithms need to satisfy for any successful application. One has to do with guarantees for convergence, and the other concerns the convergence rate. The vast majority of reinforcement learning algorithms belong to a class of learning algorithms known as stochastic approximation (SA). The objective here is to review the foundations of reinforcement learning algorithm design based on recent and ancient results from SA. In particular, it was established in [Borkar and Meyn, 2000] that both stability and convergence of these algorithms are guaranteed by analyzing the stability of two associated ODEs. Moreover, if the linearized ODE passes a simple eigenvalue test, then an optimal rate of convergence is guaranteed. This chapter contains a survey of these concepts, along with a survey of the new class of Zap reinforcement learning algorithms introduced by the authors. These algorithms can achieve convergence almost universally, while also guaranteeing optimal rate of convergence.
No file

Dates and versions

hal-03541756 , version 1 (24-01-2022)

Identifiers

Cite

Adithya Devraj, Ana Bušić, Sean Meyn. Fundamental Design Principles for Reinforcement Learning Algorithms. Kyriakos G. Vamvoudakis; Yan Wan; Frank L. Lewis; Derya Cansever. Handbook of Reinforcement Learning and Control, 325, Springer International Publishing, pp.75-137, 2021, Studies in Systems, Decision and Control, ⟨10.1007/978-3-030-60990-0_4⟩. ⟨hal-03541756⟩
101 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More