LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue Mathematics of Operations Research Année : 2023

LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality

Résumé

We provide a framework to analyse control policies for the restless Markovian bandit model, under both finite and infinite time horizon. We show that when the population of arms goes to infinity, the value of the optimal control policy converges to the solution of a linear program (LP). We provide necessary and sufficient conditions for a generic control policy to be: i) asymptotically optimal; ii) asymptotically optimal with square root convergence rate; iii) asymptotically optimal with exponential rate. We then construct the LP-index policy that is asymptotically optimal with square root convergence rate on all models, and with exponential rate if the model is non-degenerate in finite horizon, and satisfies a uniform global attractor property in infinite horizon. We next define the LP-update policy, which is essentially a repeated LP-index policy that solves a new linear program at each decision epoch. We provide numerical experiments to compare the efficiency of LP-based policies. We compare the performance of the LP-index policy and the LP-update policy with other heuristics. Our result demonstrates that the LP-update policy outperforms the LP-index policy in general, and can have a significant advantage when the transition matrices are wrongly estimated.
Fichier principal
Vignette du fichier
LP_finite_infinite.pdf (806.54 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03262307 , version 1 (18-06-2021)
hal-03262307 , version 2 (11-05-2022)
hal-03262307 , version 3 (13-09-2022)
hal-03262307 , version 4 (21-12-2023)

Licence

Paternité

Identifiants

Citer

Nicolas Gast, Bruno Gaujal, Chen Yan. LP-based policies for restless bandits: necessary and sufficient conditions for (exponentially fast) asymptotic optimality. Mathematics of Operations Research, 2023, pp.1-29. ⟨10.1287/moor.2022.0101⟩. ⟨hal-03262307v4⟩
245 Consultations
434 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More