Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3 - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3

Résumé

In this paper we study the problem-solving ability of the Large Language Model known as GPT-3 (codename DaVinci), by considering its performance in solving tasks proposed in the "Bebras International Challenge on Informatics and Computational Thinking". In our experiment, GPT-3 was able to answer with a majority of correct answers about one third of the Bebras tasks we submitted to it. The linguistic fluency of GPT-3 is impressive and, at a first reading, its explanations sound coherent, on-topic and authoritative; however the answers it produced are in fact erratic and the explanations often questionable or plainly wrong. The tasks in which the system performs better are those that describe a procedure, asking to execute it on a specific instance of the problem. Tasks solvable with simple, one-step deductive reasoning are more likely to obtain better answers and explanations. Synthesis tasks, or tasks that require a more complex logical consistency get the most incorrect answers.
Fichier principal
Vignette du fichier
csedu.pdf (150.66 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY NC ND - Paternité - Pas d'utilisation commerciale - Pas de modification

Dates et versions

hal-04343267 , version 1 (13-12-2023)

Licence

Paternité - Pas d'utilisation commerciale - Pas de modification

Identifiants

Citer

Carlo Bellettini, Michael Lodi, Violetta Lonati, Mattia Monga, Anna Morpurgo. Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. CSEDU 2023 - 15th International Conference on Computer Supported Education, Apr 2023, Prague, Czech Republic. pp.59-69, ⟨10.5220/0012007500003470⟩. ⟨hal-04343267⟩
17 Consultations
13 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More