MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic

Résumé

Theory of Mind (ToM) is a critical component of intelligence, yet accurately measuring it continues to be a subject of debate. Prior research has attempted to apply human ToM assessments to natural language processing models using either human-created standardized tests or rule-based templates. However, these methods primarily focus on simplistic reasoning and require further validation. In this study, we utilize dynamic epistemic logic, which has established overlaps with ToM, to generate more intricate problems. We also introduce novel verbalization techniques to express these problems using natural language. Our findings indicate that certain language model scaling (from 70M to 6B and 350M to 174B) does not consistently yield results better than random chance. While GPT-4 demonstrates improved epistemic reasoning capabilities, there is still room for enhancement. Our code and datasets are publicly available https://github.com/antoinelrnld/modlog https://huggingface.co/datasets/sileod/mindgames
Fichier principal
Vignette du fichier
mindgames.pdf (164.61 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY - Paternité

Dates et versions

hal-04098588 , version 1 (16-05-2023)

Identifiants

Citer

Damien Sileo, Antoine Lernould. MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic. 2023. ⟨hal-04098588⟩
22 Consultations
49 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More