Long-Term Values in Markov Decision Processes, (Co)Algebraically - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2018

Long-Term Values in Markov Decision Processes, (Co)Algebraically


This paper studies Markov decision processes (MDPs) from the categorical perspective of coalgebra and algebra. Probabilistic systems, similar to MDPs but without rewards, have been extensively studied, also coalgebraically, from the perspective of program semantics. In this paper, we focus on the role of MDPs as models in optimal planning, where the reward structure is central. The main contributions of this paper are (i) to give a coinductive explanation of policy improvement using a new proof principle, based on Banach’s Fixpoint Theorem, that we call contraction coinduction, and (ii) to show that the long-term value function of a policy with respect to discounted sums can be obtained via a generalized notion of corecursive algebra, which is designed to take boundedness into account. We also explore boundedness features of the Kantorovich lifting of the distribution monad to metric spaces.
Fichier principal
Vignette du fichier
473364_1_En_6_Chapter.pdf (475.04 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-02044650 , version 1 (21-02-2019)





Frank Feys, Helle Hvid Hansen, Lawrence S. Moss. Long-Term Values in Markov Decision Processes, (Co)Algebraically. 14th International Workshop on Coalgebraic Methods in Computer Science (CMCS), Apr 2018, Thessaloniki, Greece. pp.78-99, ⟨10.1007/978-3-030-00389-0_6⟩. ⟨hal-02044650⟩
76 View
161 Download



Gmail Facebook Twitter LinkedIn More