XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words

Abstract

Due to the absence of explicit word boundaries in the speech stream, the task of segmenting spoken sentences into word units without text supervision is particularly challenging. In this work, we leverage the most recent selfsupervised speech models that have proved to quickly adapt to new tasks through fine-tuning, even in low resource conditions. Taking inspiration from semi-supervised learning, we finetune an XLS-R model to predict word boundaries themselves produced by top-tier speech segmentation systems: DPDP, VG-HuBERT, GradSeg and DP-Parse. Once XLS-R is finetuned, it is used to infer new word boundary labels that are used in turn for another finetuning step. Our method consistently improves the performance of each system and sets a new state-of-the-art that is, on average 130% higher than the previous one as measured by the F1 score on correctly discovered word tokens on five corpora featuring different languages. Finally, our system can segment speech from languages unseen during fine-tuning in a zero-shot fashion 1 .
Fichier principal
Vignette du fichier
111_xls_r_fine_tuning_on_noisy_wor.pdf (254.56 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Licence

Dates and versions

hal-04398496 , version 1 (16-01-2024)

Licence

Identifiers

  • HAL Id : hal-04398496 , version 1

Cite

Robin Algayres, Pablo Diego-Simon, Benoît Sagot, Emmanuel Dupoux. XLS-R fine-tuning on noisy word boundaries for unsupervised speech segmentation into words. EMNLP 2023 - Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Dec 2023, Singapore, Singapore. ⟨hal-04398496⟩
57 View
17 Download

Share

Gmail Mastodon Facebook X LinkedIn More