Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret

Jean Tarbouriech
  • Function : Author
  • PersonId : 1105513
Runlong Zhou
  • Function : Author
  • PersonId : 1120365
Matteo Pirotta
  • Function : Author
  • PersonId : 1105514
Michal Valko
  • Function : Author
  • PersonId : 1120367
Alessandro Lazaric
  • Function : Author
  • PersonId : 1105515

Abstract

We study the problem of learning in the stochastic shortest path (SSP) setting, where an agent seeks to minimize the expected cost accumulated before reaching a goal state. We design a novel model-based algorithm EB-SSP that carefully skews the empirical transitions and perturbs the empirical costs with an exploration bonus to induce an optimistic SSP problem whose associated value iteration scheme is guaranteed to converge. We prove that EB-SSP achieves the minimax regret rate O(B* √ SAK), where K is the number of episodes, S is the number of states, A is the number of actions, and B* bounds the expected cumulative cost of the optimal policy from any state, thus closing the gap with the lower bound. Interestingly, EB-SSP obtains this result while being parameter-free, i.e., it does not require any prior knowledge of B*, nor of T*, which bounds the expected time-to-goal of the optimal policy from any state. Furthermore, we illustrate various cases (e.g., positive costs, or general costs when an order-accurate estimate of T* is available) where the regret only contains a logarithmic dependence on T*, thus yielding the first (nearly) horizon-free regret bound beyond the finite-horizon MDP setting.
Fichier principal
Vignette du fichier
Stochastic Shortest Path.pdf (676.49 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03479782 , version 1 (14-12-2021)

Identifiers

  • HAL Id : hal-03479782 , version 1

Cite

Jean Tarbouriech, Runlong Zhou, Simon S Du, Matteo Pirotta, Michal Valko, et al.. Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret. Neural Information Processing Systems (NeurIPS), Dec 2021, Virtual/Sydney, Australia. ⟨hal-03479782⟩
19 View
37 Download

Share

Gmail Facebook X LinkedIn More