SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Résumé

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations (S2ST) mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on Europarl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pretraining and sparse scaling using Mixture-of-Experts bring large gains to translation performance. We are open-sourcing the mined data, speech encoders used for mining, multilingual HuBERT models in four language families for target unit generation, languagespecific vocoders for speech synthesis from discrete units, and S2S models trained and presented in this work. 1
Fichier principal
Vignette du fichier
Large_Scale_Speech_to_Speech_Mining-2.pdf (464.58 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04264040 , version 1 (29-10-2023)

Licence

Paternité

Identifiants

Citer

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, et al.. SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations. ACL 2023 - 61st Annual Meeting of the Association for Computational Linguistics, Aug 2023, Toronto, Canada. ⟨10.18653/v1/2023.acl-long.899⟩. ⟨hal-04264040⟩
10 Consultations
4 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More