Communication Dans Un Congrès Année : 2025

Identifying Rare Languages in Common Crawl Data is a Needles-in-a-Haystack Problem

Résumé

Automatic language identification is frequently framed as a multi-class classification problem. However, when creating digital corpora for less commonly written languages, it may be more appropriate to consider it a data mining problem. For these varieties, one knows ahead of time that the vast majority of documents are of little interest. By minimizing resources spent on classifying such documents, we can create corpora covering previously overlooked languages faster than existing pipelines. To demonstrate the effectiveness of the targeted mining perspective, we introduce a new pipeline that can filter a single snapshot in two hours. We also provide web corpora for several French-based Creoles.

Fichier principal
Vignette du fichier
2025.findings-emnlp.77.pdf (216.04 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05361348 , version 1 (12-11-2025)

Licence

Identifiants

Citer

Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot. Identifying Rare Languages in Common Crawl Data is a Needles-in-a-Haystack Problem. Findings of the Association for Computational Linguistics: EMNLP 2025, Nov 2025, Suzhou, China. pp.1460-1473, ⟨10.18653/v1/2025.findings-emnlp.77⟩. ⟨hal-05361348⟩
184 Consultations
275 Téléchargements

Altmetric

Partager

  • More