Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3 - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3

Abstract

In this paper we study the problem-solving ability of the Large Language Model known as GPT-3 (codename DaVinci), by considering its performance in solving tasks proposed in the "Bebras International Challenge on Informatics and Computational Thinking". In our experiment, GPT-3 was able to answer with a majority of correct answers about one third of the Bebras tasks we submitted to it. The linguistic fluency of GPT-3 is impressive and, at a first reading, its explanations sound coherent, on-topic and authoritative; however the answers it produced are in fact erratic and the explanations often questionable or plainly wrong. The tasks in which the system performs better are those that describe a procedure, asking to execute it on a specific instance of the problem. Tasks solvable with simple, one-step deductive reasoning are more likely to obtain better answers and explanations. Synthesis tasks, or tasks that require a more complex logical consistency get the most incorrect answers.
Fichier principal
Vignette du fichier
csedu.pdf (150.66 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Licence : CC BY NC ND - Attribution - NonCommercial - NoDerivatives

Dates and versions

hal-04343267 , version 1 (13-12-2023)

Licence

Attribution - NonCommercial - NoDerivatives

Identifiers

Cite

Carlo Bellettini, Michael Lodi, Violetta Lonati, Mattia Monga, Anna Morpurgo. Davinci Goes to Bebras: A Study on the Problem Solving Ability of GPT-3. CSEDU 2023 - 15th International Conference on Computer Supported Education, Apr 2023, Prague, Czech Republic. pp.59-69, ⟨10.5220/0012007500003470⟩. ⟨hal-04343267⟩
20 View
14 Download

Altmetric

Share

Gmail Facebook X LinkedIn More