Power Mean Estimation in Stochastic Monte-Carlo Tree Search - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2024

Power Mean Estimation in Stochastic Monte-Carlo Tree Search

Résumé

Monte-Carlo Tree Search (MCTS) is a widelyused strategy for online planning that combines Monte-Carlo sampling with forward tree search. Its success relies on the Upper Confidence bound for Trees (UCT) algorithm, an extension of the UCB method for multi-arm bandits. However, the theoretical foundation of UCT is incomplete due to an error in the logarithmic bonus term for action selection, leading to the development of Fixed-Depth-MCTS with a polynomial exploration bonus to balance exploration and exploitation [Shah et al., 2022]. Both UCT and Fixed-Depth-MCTS suffer from biased value estimation: the weighted sum underestimates the optimal value, while the maximum valuation overestimates it [Coulom, 2006]. The power mean estimator offers a balanced solution, lying between the average and maximum values. Power-UCT [Dam et al., 2019] incorporates this estimator for more accurate value estimates but its theoretical analysis remains incomplete. This paper introduces Stochastic-Power-UCT, an MCTS algorithm using the power mean estimator and tailored for stochastic MDPs. We analyze its polynomial convergence in estimating root node values and show that it shares the same convergence rate of O(n -1/2 ), with n is the number of visited trajectories, as Fixed-Depth-MCTS, with the latter being a special case of the former. Our theoretical results are validated with empirical tests across various stochastic MDP environments.

Fichier principal
Vignette du fichier
607_Power_Mean_Estimation_in_S.pdf (1.39 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04714124 , version 1 (30-09-2024)

Licence

Identifiants

  • HAL Id : hal-04714124 , version 1

Citer

Tuan Quang Tuan Dam, Odalric-Ambrym Maillard, Emilie Kaufmann. Power Mean Estimation in Stochastic Monte-Carlo Tree Search. Uncertainty in Artificial Intelligence, Jul 2024, Barcelona, Spain. ⟨hal-04714124⟩
15 Consultations
4 Téléchargements

Partager

More