Symphony of experts: orchestration with adversarial insights in reinforcement learning - Inria - Institut national de recherche en sciences et technologies du numérique
Preprints, Working Papers, ... Year : 2023

Symphony of experts: orchestration with adversarial insights in reinforcement learning

Abstract

Structured reinforcement learning leverages policies with advantageous properties to reach better performance, particularly in scenarios where exploration poses challenges. We explore this field through the concept of orchestration, where a (small) set of expert policies guides decision-making; the modeling thereof constitutes our first contribution. We then establish value-functions regret bounds for orchestration in the tabular setting by transferring regret-bound results from adversarial settings. We generalize and extend the analysis of natural policy gradient in Agarwal et al. [2021, Section 5.3] to arbitrary adversarial aggregation strategies. We also extend it to the case of estimated advantage functions, providing insights into sample complexity both in expectation and high probability. A key point of our approach lies in its arguably more transparent proofs compared to existing methods. Finally, we present simulations for a stochastic matching toy model.
Fichier principal
Vignette du fichier
Agreg-RL--JMS.pdf (467.66 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04256705 , version 1 (24-10-2023)

Identifiers

Cite

Matthieu Jonckheere, Chiara Mignacco, Gilles Stoltz. Symphony of experts: orchestration with adversarial insights in reinforcement learning. 2023. ⟨hal-04256705⟩
130 View
65 Download

Altmetric

Share

More