Threats of a replication crisis in empirical computer science - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Communications of the ACM Year : 2020

Threats of a replication crisis in empirical computer science

Andy Cockburn
  • Function : Author
  • PersonId : 1005483
Pierre Dragicevic
Lonni Besançon


Many areas of computer science research (e.g., performance analysis, software engineering, artificial intelligence, and human-computer interaction) validate research claims by using statistical significance as the standard of evidence. A loss of confidence in statistically significant findings is plaguing other empirical disciplines, yet there has been relatively little debate of this issue and its associated 'replication crisis' in computer science. We review factors that have contributed to the crisis in other disciplines, with a focus on problems stemming from an over-reliance on-and misuse of-null hypothesis significance testing. Computer science research can be greatly improved by following the steps taken by other disciplines, such as using more sophisticated evidentiary criteria, and showing greater openness and transparency through experimental preregistration and data/artifact repositories.
Fichier principal
Vignette du fichier
Computer_Science_Research_and_the_Replication_Crisis_preprint.pdf (394.23 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-02907143 , version 1 (27-07-2020)



Andy Cockburn, Pierre Dragicevic, Lonni Besançon, Carl Gutwin. Threats of a replication crisis in empirical computer science. Communications of the ACM, 2020, 63 (8), pp.70-79. ⟨10.1145/3360311⟩. ⟨hal-02907143⟩
208 View
520 Download



Gmail Mastodon Facebook X LinkedIn More