Improved Regret Bounds for Thompson Sampling in Linear Quadratic Control Problems - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue Proceedings of Machine Learning Research Année : 2018

Improved Regret Bounds for Thompson Sampling in Linear Quadratic Control Problems

Marc Abeille
  • Fonction : Auteur
  • PersonId : 1072166
Alessandro Lazaric

Résumé

Thompson sampling (TS) is an effective approach to trade off exploration and exploration in reinforcement learning. Despite its empirical success and recent advances, its theoretical analysis is often limited to the Bayesian setting, finite state-action spaces, or finite-horizon problems. In this paper, we study an instance of TS in the challenging setting of the infinite-horizon linear quadratic (LQ) control, which models problems with continuous state-action variables, linear dynamics, and quadratic cost. In particular, we analyze the regret in the frequentist sense (i.e., for a fixed unknown environment) in one-dimensional systems. We derive the first O(√ T) frequentist regret bound for this problem, thus significantly improving the O(T 2/3) bound of Abeille & Lazaric (2017) and matching the frequentist performance derived by Abbasi-Yadkori & Szepesvári (2011) for an optimistic approach and the Bayesian result of Ouyang et al. (2017). We obtain this result by developing a novel bound on the regret due to policy switches, which holds for LQ systems of any dimensionality and it allows updating the parameters and the policy at each step, thus overcoming previous limitations due to lazy updates. Finally, we report numerical simulations supporting the conjecture that our result extends to multi-dimensional systems.

Domaines

Autres [stat.ML]
Fichier principal
Vignette du fichier
abeille18a-supp.pdf (389.24 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02767726 , version 1 (04-06-2020)

Identifiants

  • HAL Id : hal-02767726 , version 1

Citer

Marc Abeille, Alessandro Lazaric. Improved Regret Bounds for Thompson Sampling in Linear Quadratic Control Problems. Proceedings of Machine Learning Research, 2018, Proceedings of ICML 2018, 80. ⟨hal-02767726⟩
56 Consultations
71 Téléchargements

Partager

Gmail Facebook X LinkedIn More