Decentralized model-free reinforcement learning in stochastic games with average-reward objective - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Decentralized model-free reinforcement learning in stochastic games with average-reward objective

Résumé

We propose the first model-free algorithm that achieves low regret performance for decentralized learning in two-player zerosum tabular stochastic games with infinite-horizon average-reward objective. In decentralized learning, the learning agent controls only one player and tries to achieve low regret performances against an arbitrary opponent. This contrasts with centralized learning where the agent tries to approximate the Nash equilibrium by controlling both players. In our infinite-horizon undiscounted setting, additional structure assumptions is needed to provide good behaviors of learning processes : here we assume for every strategy of the opponent, the agent has a way to go from any state to any other. This assumption is the analogous to the "communicating" assumption in the MDP setting. We show that our Decentralized Optimistic Nash Q-Learning (DONQ-learning) algorithm achieves both sublinear high probability regret of order 3/4 and sublinear expected regret of order 2/3. Moreover, our algorithm enjoys a low computational complexity and low memory space requirement compared to the previous works of [23] and [9] in the same setting.
Fichier principal
Vignette du fichier
2301.05630.pdf (272.56 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04161628 , version 1 (13-07-2023)

Licence

Paternité

Identifiants

  • HAL Id : hal-04161628 , version 1

Citer

Romain Cravic, Nicolas Gast, Bruno Gaujal. Decentralized model-free reinforcement learning in stochastic games with average-reward objective. AAMAS 2023 - International Conference on Autonomous Agents and Multiagent Systems, May 2023, London (U.K.), United Kingdom. pp.1-13. ⟨hal-04161628⟩
46 Consultations
24 Téléchargements

Partager

Gmail Facebook X LinkedIn More