SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations

Abstract

We present SpeechMatrix, a large-scale multilingual corpus of speech-to-speech translations (S2ST) mined from real speech of European Parliament recordings. It contains speech alignments in 136 language pairs with a total of 418 thousand hours of speech. To evaluate the quality of this parallel speech, we train bilingual speech-to-speech translation models on mined data only and establish extensive baseline results on Europarl-ST, VoxPopuli and FLEURS test sets. Enabled by the multilinguality of SpeechMatrix, we also explore multilingual speech-to-speech translation, a topic which was addressed by few other works. We also demonstrate that model pretraining and sparse scaling using Mixture-of-Experts bring large gains to translation performance. We are open-sourcing the mined data, speech encoders used for mining, multilingual HuBERT models in four language families for target unit generation, languagespecific vocoders for speech synthesis from discrete units, and S2S models trained and presented in this work. 1
Fichier principal
Vignette du fichier
Large_Scale_Speech_to_Speech_Mining-2.pdf (464.58 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-04264040 , version 1 (29-10-2023)

Licence

Attribution

Identifiers

Cite

Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, et al.. SpeechMatrix: A Large-Scale Mined Corpus of Multilingual Speech-to-Speech Translations. ACL 2023 - 61st Annual Meeting of the Association for Computational Linguistics, Aug 2023, Toronto, Canada. ⟨10.18653/v1/2023.acl-long.899⟩. ⟨hal-04264040⟩
11 View
6 Download

Altmetric

Share

Gmail Facebook X LinkedIn More