The Impact of Removing Head Movements on Audio-visual Speech Enhancement - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2022

The Impact of Removing Head Movements on Audio-visual Speech Enhancement

Résumé

This paper investigates the impact of head movements on audiovisual speech enhancement (AVSE). Although being a common conversational feature, head movements have been ignored by past and recent studies: they challenge today's learning-based methods as they often degrade the performance of models that are trained on clean, frontal, and steady face images. To alleviate this problem, we propose to use robust face frontalization (RFF) in combination with an AVSE method based on a variational auto-encoder (VAE) model. We briefly describe the basic ingredients of the proposed pipeline and we perform experiments with a recently released audiovisual dataset. In the light of these experiments, and based on three standard metrics, namely STOI, PESQ and SI-SDR, we conclude that RFF improves the performance of AVSE by a considerable margin.
Fichier principal
Vignette du fichier
Kang-ICASSP22-CR.pdf (836.28 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03551610 , version 1 (01-02-2022)
hal-03551610 , version 2 (02-02-2022)

Identifiants

  • HAL Id : hal-03551610 , version 1

Citer

Zhiqi Kang, Mostafa Sadeghi, Radu Horaud, Xavier Alameda-Pineda, Jacob Donley, et al.. The Impact of Removing Head Movements on Audio-visual Speech Enhancement. ICASSP 2022 - International Conference on Acoustics, Speech and Signal Processing, IEEE Signal Processing Society, May 2022, Singapore, Singapore. ⟨hal-03551610v1⟩
298 Consultations
182 Téléchargements

Partager

More