SONAR: Sentence-Level Multimodal and Language-Agnostic Representations - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Preprints, Working Papers, ... Year : 2023

SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

Abstract

We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. Our encoders outperform existing speech encoders on similarity search tasks. We also provide a text decoder for 200 languages, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. Our text-to-text results are competitive compared to the state-of-the-art NLLB 1B model, despite the fixed-size bottleneck representation. Our zero-shot speech-to-text translation results compare favorably with strong supervised baselines such as Whisper.

Dates and versions

hal-04264028 , version 1 (29-10-2023)

Licence

Attribution

Identifiers

Cite

Paul-Ambroise Duquenne, Holger Schwenk, Benoît Sagot. SONAR: Sentence-Level Multimodal and Language-Agnostic Representations. 2023. ⟨hal-04264028⟩
46 View
0 Download

Altmetric

Share

Gmail Facebook X LinkedIn More