Modulation spectral features for speech emotion recognition using deep neural networks - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue Speech Communication Année : 2023

Modulation spectral features for speech emotion recognition using deep neural networks

Résumé

This work explores the use of constant-Q transform based modulation spectral features (CQT-MSF) for speech emotion recognition (SER). The human perception and analysis of sound comprise of two important cognitive parts: early auditory analysis and cortex-based processing. The early auditory analysis considers spectrogram-based representation whereas cortex-based analysis includes extraction of temporal modulations from the spectrogram. This temporal modulation representation of spectrogram is called modulation spectral feature (MSF). As the constant-Q transform (CQT) provides higher resolution at emotion salient low-frequency regions of speech, we find that CQTbased spectrogram, together with its temporal modulations, provides a representation enriched with emotion-specific information. We argue that CQT-MSF when used with a 2-dimensional convolutional network can provide a time-shift invariant and deformation insensitive representation for SER. Our results show that CQT-MSF outperforms standard mel-scale based spectrogram and its modulation features on two popular SER databases, Berlin EmoDB and RAVDESS. We also show that our proposed feature outperforms the shift and deformation invariant scattering transform coefficients, hence, showing the importance of joint hand-crafted and self-learned feature extraction instead of reliance on complete hand-crafted features. Finally, we perform Grad-CAM analysis to visually inspect the contribution of constant-Q modulation features over SER.
Fichier principal
Vignette du fichier
CQT_MSF_HAL.pdf (2.36 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03862222 , version 1 (21-11-2022)

Identifiants

Citer

Premjeet Singh, Md Sahidullah, Goutam Saha. Modulation spectral features for speech emotion recognition using deep neural networks. Speech Communication, 2023, 146 (January), pp.53-69. ⟨10.1016/j.specom.2022.11.005⟩. ⟨hal-03862222⟩
40 Consultations
189 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More