AVATAR: Unconstrained Audiovisual Speech Recognition - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

AVATAR: Unconstrained Audiovisual Speech Recognition


Audiovisual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth. Unlike works that simply focus on the lip motion, we investigate the contribution of entire visual frames (visual actions, objects, background etc.). This is particularly useful for unconstrained videos, where the speaker is not necessarily visible. To solve this task, we propose a new sequence-to-sequence AudioVisual ASR TrAnsformeR (AVATAR) which is trained end-to-end from spectrograms and full-frame RGB. To prevent the audio stream from dominating training, we propose different word-masking strategies, thereby encouraging our model to pay attention to the visual stream. We demonstrate the contribution of the visual modality on the How2 AV-ASR benchmark, especially in the presence of simulated noise, and show that our model outperforms all other prior work by a large margin. Finally, we also create a new, realworld test bed for AV-ASR called VisSpeech, which demonstrates the contribution of the visual modality under challenging audio conditions.
Fichier principal
Vignette du fichier
avalar_visspeech.pdf (10.57 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03717330 , version 1 (08-07-2022)


  • HAL Id : hal-03717330 , version 1


Valentin Gabeur, Paul Hongsuck Seo, Arsha Nagrani, Chen Sun, Karteek Alahari, et al.. AVATAR: Unconstrained Audiovisual Speech Recognition. INTERSPEECH 2022 - Conference of the International Speech Communication Association, Sep 2022, Incheon, South Korea. pp.1-6. ⟨hal-03717330⟩
50 View
18 Download


Gmail Facebook X LinkedIn More