First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT

Abstract

Multilingual pretrained language models have demonstrated remarkable zero-shot crosslingual transfer capabilities. Such transfer emerges by fine-tuning on a task of interest in one language and evaluating on a distinct language, not seen during the fine-tuning. Despite promising results, we still lack a proper understanding of the source of this transfer. Using a novel layer ablation technique and analyses of the model's internal representations, we show that multilingual BERT, a popular multilingual language model, can be viewed as the stacking of two sub-networks: a multilingual encoder followed by a taskspecific language-agnostic predictor. While the encoder is crucial for cross-lingual transfer and remains mostly unchanged during finetuning, the task predictor has little importance on the transfer and can be reinitialized during fine-tuning. We present extensive experiments with three distinct tasks, seventeen typologically diverse languages and multiple domains to support our hypothesis.
Fichier principal
Vignette du fichier
2021.eacl-main.189.pdf (766.53 Ko) Télécharger le fichier
Origin : Publisher files allowed on an open archive

Dates and versions

hal-03239087 , version 1 (27-05-2021)

Identifiers

  • HAL Id : hal-03239087 , version 1

Cite

Benjamin Muller, Yanai Elazar, Benoît Sagot, Djamé Seddah. First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT. EACL 2021 - The 16th Conference of the European Chapter of the Association for Computational Linguistics, Apr 2021, Kyiv / Virtual, Ukraine. ⟨hal-03239087⟩
108 View
153 Download

Share

Gmail Facebook X LinkedIn More