Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Transactions of the Association for Computational Linguistics Year : 2022

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Isaac Caswell
  • Function : Author
Lisa Wang
  • Function : Author
Ahsan Wahab
  • Function : Author
Daan van Esch
  • Function : Author
Artem Sokolov
  • Function : Author
Monang Setyawan
  • Function : Author
Clara Rivera
  • Function : Author
Iroro Orife
  • Function : Author
Nanda Muhammad
  • Function : Author
Ayanda Mnyakeni
  • Function : Author
Colin Leong
  • Function : Author
Nze Lawson
  • Function : Author
Sneha Kudugunta
  • Function : Author
Sakhile Dlamini
  • Function : Author
Sakine Çabuk Balli
  • Function : Author
Stella Biderman
  • Function : Author
Ankur Bapna
  • Function : Author
Pallavi Baljekar
  • Function : Author

Abstract

With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. However, to date there has been no systematic analysis of the quality of these publicly available datasets, or whether the datasets actually contain content in the languages they claim to represent. In this work, we manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4), and audit the correctness of language codes in a sixth (JW300). We find that lower-resource corpora have systematic issues: at least 15 corpora are completely erroneous, and a significant fraction contains less than 50% sentences of acceptable quality. Similarly, we find 82 corpora that are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-speakers of the languages in question, and supplement the human judgements with automatic analyses. Inspired by our analysis, we recommend techniques to evaluate and improve multilingual corpora and discuss the risks that come with low-quality data releases.
Fichier principal
Vignette du fichier
tacl_a_00447.pdf (348.24 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03177623 , version 1 (13-02-2022)

Licence

Attribution

Identifiers

Cite

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, et al.. Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 2022, 10, pp.50-72. ⟨10.1162/tacl_a_00447⟩. ⟨hal-03177623⟩
327 View
122 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More