On the Scaling Laws of Geographical Representation in Language Models - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

On the Scaling Laws of Geographical Representation in Language Models

Résumé

Language models have long been shown to embed geographical information in their hidden representations. This line of work has recently been revisited by extending this result to Large Language Models (LLMs). In this paper, we propose to fill the gap between well-established and recent literature by observing how geographical knowledge evolves when scaling language models. We show that geographical knowledge is observable even for tiny models, and that it scales consistently as we increase the model size. Notably, we observe that larger language models cannot mitigate the geographical bias that is inherent to the training data.

Dates et versions

hal-04552549 , version 1 (19-04-2024)

Licence

Identifiants

Citer

Nathan Godey, Eric Villemonte de La Clergerie, Benoît Sagot. On the Scaling Laws of Geographical Representation in Language Models. LREC-Coling 2024 - Joint International Conference on Computational Linguistics, Language Resources and Evaluation, May 2024, Torino, Italy. pp.12416-12422. ⟨hal-04552549⟩
21 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More