SPICE+: Evaluation of automatic audio captioning systems with pre-trained language models - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

SPICE+: Evaluation of automatic audio captioning systems with pre-trained language models

Abstract

Audio captioning aims at describing acoustic scenes with natural language. Systems are currently evaluated by image cap- tioning metrics CIDEr and SPICE. However, recent studies have highlighted a poor correlation of these metrics with human assessments. In this paper, we propose SPICE+, a modification of SPICE that improves caption annotation and comparison with pre-trained language models. The metric parses captions to semantic graphs with a deep dependency annotation model and a refined set of linguistic rules, then compares sentence embeddings of candidate and reference semantic elements. We formulate a score for general-purpose captioning evaluation, that can be tailored to more specific applications. Combined with fluency error detection, the metric achieves competitive performance on the FENSE benchmark, with 84.0% accuracy on AudioCaps and 74.1% on Clotho. Further experiments show that the metric behaves similarly to the full sentence embedding similarity, while the decomposition into semantic elements allows better interpretability of scores and can provide additional information on the properties of captioning systems.
Fichier principal
Vignette du fichier
icassp2023spicep.pdf (186.24 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03933981 , version 1 (11-01-2023)

Licence

Identifiers

  • HAL Id : hal-03933981 , version 1

Cite

Félix Gontier, Romain Serizel, Christophe Cerisara. SPICE+: Evaluation of automatic audio captioning systems with pre-trained language models. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), Jun 2023, Rhodes Island, Greece. ⟨hal-03933981⟩
74 View
156 Download

Share

Gmail Mastodon Facebook X LinkedIn More