Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2020

Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures

Abstract

Communicative gestures and speech prosody are tightly linked. Our aim is to predict when gestures are performed based on prosody. We develop a model based on a seq2seq recurrent neural network with attention mechanism. The model is trained on a corpus of natural dyadic interaction where the speech prosody and the gestures have been annotated. Because the output of the model is a sequence, we use a sequence comparison technique to evaluate the model performance. We find that the model can predict certain gesture classes. In our experiment, we also replace some input features with random values to find which prosody features are pertinent. We find that the F0 is pertinent. Lastly, we also train the model on one speaker and test it with the other speaker to find whether the model is generalisable. We find that the models which we train on one speaker also works for another speaker of the same conversation.
Fichier principal
Vignette du fichier
wacai_2020_7_.pdf (489.48 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-02933487 , version 1 (08-09-2020)

Identifiers

  • HAL Id : hal-02933487 , version 1

Cite

Fajrian Yunus, Chloé Clavel, Catherine I Pelachaud. Sequence-to-Sequence Predictive models: from Prosody to Communicative Gestures. Workshop sur les Affects, Compagnons artificiels et Interactions, CNRS, Université Toulouse Jean Jaurès, Université de Bordeaux, Jun 2020, Saint Pierre d'Oléron, France. ⟨hal-02933487⟩
139 View
148 Download

Share

Gmail Facebook X LinkedIn More