Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3 - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3

Résumé

In this paper we study the problem-solving ability of the Large Language Model known as GPT-3 (codename DaVinci), by considering its performance in solving tasks proposed in the "Bebras International Challenge on Informatics and Computational Thinking". In our experiment, GPT-3 was able to answer with a majority of correct answers about one third of the Bebras tasks we submitted to it. The linguistic fluency of GPT-3 is impressive and, at a first reading, its explanations sound coherent, on-topic and authoritative; however the answers it produced are in fact erratic and the explanations often questionable or plainly wrong. The tasks in which the system performs better are those that describe a procedure, asking to execute it on a specific instance of the problem. Tasks solvable with simple, one-step deductive reasoning are more likely to obtain better answers and explanations. Synthesis tasks, or tasks that require a more complex logical consistency get the most incorrect answers.
Fichier principal
Vignette du fichier
csedu.pdf (150.66 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04343267 , version 1 (13-12-2023)

Licence

Identifiants

Citer

Carlo Bellettini, Michael Lodi, Violetta Lonati, Mattia Monga, Anna Morpurgo. Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. CSEDU 2023 - 15th International Conference on Computer Supported Education, Apr 2023, Prague, Czech Republic. pp.59-69, ⟨10.5220/0012007500003470⟩. ⟨hal-04343267⟩
43 Consultations
28 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More