PAGnol: An Extra-Large French Generative Model - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Rapport (Rapport De Recherche) Année : 2021

PAGnol: An Extra-Large French Generative Model

Résumé

Access to large pre-trained models of varied architectures, in many different languages, is central to the democratization of NLP. We introduce PAGnol, a collection of French GPT models. Using scaling laws, we efficiently train PAGnol-XL (1.5B parameters) with the same computational budget as CamemBERT, a model 13 times smaller. PAGnol-XL is the largest model trained to date for the French language. We plan to train increasingly large and performing versions of PAGnol, exploring the capabilities of French extreme-scale models. For this first release, we focus on the pre-training and scaling calculations underlining PAGnol. We fit a scaling law for compute for the French language, and compare it with its English counterpart. We find the pre-training dataset significantly conditions the quality of the outputs, with common datasets such as OSCAR leading to low-quality offensive text. We evaluate our models on discriminative and generative tasks in French, comparing to other state-of-the-art French and multilingual models, and reaching the state of the art in the abstract summarization task. Our research was conducted on the public GENCI Jean Zay supercomputer, and our models up to the Large are made publicly available.

Dates et versions

hal-03540159 , version 1 (23-01-2022)

Identifiants

Citer

Julien Launay, Elena Tommasone, Baptiste Pannier, François Boniface, Amélie Chatelain, et al.. PAGnol: An Extra-Large French Generative Model. [Research Report] LightON. 2021. ⟨hal-03540159⟩
100 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More