When zero may not be zero: A cautionary note on the use of inter‐rater reliability in evaluating grant peer review - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Journal of the Royal Statistical Society: Series A Statistics in Society Year : 2021

When zero may not be zero: A cautionary note on the use of inter‐rater reliability in evaluating grant peer review

Abstract

Considerable attention has focused on studying reviewer agreement via inter-rater reliability (IRR) as a way to assess the quality of the peer review process. Inspired by a recent study that reported an IRR of zero in the mock peer review of top-quality grant proposals, we use real data from a com- plete range of submissions to the National Institutes of Health and to the American Institute of Biological Sciences to bring awareness to two important issues with using IRR for assess- ing peer review quality. First, we demonstrate that estimat- ing local IRR from subsets of restricted-quality proposals will likely result in zero estimates under many scenarios. In both data sets, we find that zero local IRR estimates are more likely when subsets of top-quality proposals rather than bottom-quality proposals are considered. However, zero esti- mates from range-restricted data should not be interpreted as indicating arbitrariness in peer review. On the contrary, de- spite different scoring scales used by the two agencies, when complete ranges of proposals are considered, IRR estimates are above 0.6 which indicates good reviewer agreement. Furthermore, we demonstrate that, with a small number of reviewers per proposal, zero estimates of IRR are possible even when the true value is not zero.
Fichier principal
Vignette du fichier
ErosMartLee-JRSS-A-2021.pdf (553.69 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03522263 , version 1 (12-01-2022)

Identifiers

Cite

Elena Erosheva, Patrícia Martinková, Carole Lee. When zero may not be zero: A cautionary note on the use of inter‐rater reliability in evaluating grant peer review. Journal of the Royal Statistical Society: Series A Statistics in Society, 2021, 184, pp.904 - 919. ⟨10.1111/rssa.12681⟩. ⟨hal-03522263⟩
46 View
169 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More