SPICE+: Evaluation of automatic audio captioning systems with pre-trained language models - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

SPICE+: Evaluation of automatic audio captioning systems with pre-trained language models

Résumé

Audio captioning aims at describing acoustic scenes with natural language. Systems are currently evaluated by image cap- tioning metrics CIDEr and SPICE. However, recent studies have highlighted a poor correlation of these metrics with human assessments. In this paper, we propose SPICE+, a modification of SPICE that improves caption annotation and comparison with pre-trained language models. The metric parses captions to semantic graphs with a deep dependency annotation model and a refined set of linguistic rules, then compares sentence embeddings of candidate and reference semantic elements. We formulate a score for general-purpose captioning evaluation, that can be tailored to more specific applications. Combined with fluency error detection, the metric achieves competitive performance on the FENSE benchmark, with 84.0% accuracy on AudioCaps and 74.1% on Clotho. Further experiments show that the metric behaves similarly to the full sentence embedding similarity, while the decomposition into semantic elements allows better interpretability of scores and can provide additional information on the properties of captioning systems.
Fichier principal
Vignette du fichier
icassp2023spicep.pdf (186.24 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03933981 , version 1 (11-01-2023)

Licence

Paternité

Identifiants

  • HAL Id : hal-03933981 , version 1

Citer

Félix Gontier, Romain Serizel, Christophe Cerisara. SPICE+: Evaluation of automatic audio captioning systems with pre-trained language models. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), Jun 2023, Rhodes Island, Greece. ⟨hal-03933981⟩
65 Consultations
122 Téléchargements

Partager

Gmail Facebook X LinkedIn More