Kernel-based reinforcement Learning: A finite-time analysis - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

Kernel-based reinforcement Learning: A finite-time analysis

Omar D Domingues
  • Function : Author
Matteo Pirotta
  • Function : Author
  • PersonId : 1023840
Michal Valko

Abstract

We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning problems whose state-action space is endowed with a metric. We introduce Kernel-UCBVI, a model-based optimistic algorithm that leverages the smoothness of the MDP and a non-parametric kernel estimator of the rewards and transitions to efficiently balance exploration and exploitation. Unlike existing approaches with regret guarantees, it does not use any kind of partitioning of the state-action space. For problems with $K$ episodes and horizon $H$, we provide a regret bound of O H 3 K max(1 2 , 2d 2d+1) , where $d$ is the covering dimension of the joint state-action space. This is the first regret bound for kernel-based RL using smoothing kernels, which requires very weak assumptions on the MDP and has been previously applied to a wide range of tasks. We empirically validate our approach in continuous MDPs with sparse rewards.
Fichier principal
Vignette du fichier
domingues2021kernel-based.pdf (5.23 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-02541790 , version 1 (14-04-2020)
hal-02541790 , version 2 (16-07-2021)

Identifiers

Cite

Omar D Domingues, Pierre Ménard, Matteo Pirotta, Emilie Kaufmann, Michal Valko. Kernel-based reinforcement Learning: A finite-time analysis. International Conference on Machine Learning, Jul 2021, Vienna / Virtual, Austria. ⟨hal-02541790v2⟩
177 View
270 Download

Altmetric

Share

Gmail Facebook X LinkedIn More