Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus

Abstract

Since the introduction of large language models in Natural Language Processing, large raw corpora have played a crucial role in Computational Linguistics. However, most of these large raw corpora are either available only for English or not available to the general public due to copyright issues. Nevertheless, there are some examples of freely available multilingual corpora for training Deep Learning NLP models, such as the OSCAR and Paracrawl corpora. However, they have quality issues, especially for low-resource languages. Moreover, recreating or updating these corpora is very complex. In this work, we try to reproduce and improve the goclassy pipeline used to create the OSCAR corpus. We propose a new pipeline that is faster, modular, parameterizable, and well documented. We use it to create a corpus similar to OSCAR but larger and based on recent data.Also, unlike OSCAR, the metadata information is at the document level. We release our pipeline under an open source license and publish the corpus under a research-only license.
Fichier principal
Vignette du fichier
Ungoliant___CMLC_9-2.pdf (236.68 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03301590 , version 1 (27-07-2021)

Licence

Identifiers

Cite

Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, Benoît Sagot. Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus. CMLC 2021 - 9th Workshop on Challenges in the Management of Large Corpora, Jul 2021, Limerick / Virtual, Ireland. ⟨10.14618/ids-pub-10468⟩. ⟨hal-03301590⟩
276 View
345 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More