Hierarchical Multi-Label Propagation using Speaking Face Graphs for Multimodal Person Discovery - Inria - Institut national de recherche en sciences et technologies du numérique
Journal Articles Multimedia Tools and Applications Year : 2020

Hierarchical Multi-Label Propagation using Speaking Face Graphs for Multimodal Person Discovery

Abstract

TV archives are growing in size so fast that manually indexing becomes unfeasible. Automatic indexing techniques can be applied to overcome this issue, and this work proposes an unsupervised technique for multimodal person discovery. To achieve this goal, we propose a hierarchical label propagation technique based on quasi-flat zones theory, that learns from labeled and unlabeled data and propagates names through a multimodal graph representation. In this representation, we combine audio, video, and text processing techniques to model the data as a graph of speaking faces. In the proposed mod-eling, we extract names via optical character recognition and propagate them through the graph using audiovisual relationships between speaking faces. We also use a random walk label propagation and two graph clustering strategies to serve as baselines. The proposed label propagation techniques always outper-form the clustering baselines on the quantitative assessments. Our approach also outperforms all literature methods tested on the same dataset except for one, which uses a different preprocessing step. The proposed hierarchical label propagation and the random walk baseline produce highly equivalent results according to the Kappa coefficient, but the hierarchical propagation is parameter-free and over 9 times faster than the random walk under the same configurations.
Fichier principal
Vignette du fichier
MTAP-MPD.pdf (1.03 Mo) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

hal-02926035 , version 1 (31-08-2020)

Identifiers

Cite

Gabriel Barbosa da Fonseca, Gabriel Sargent, Ronan Sicre, Zenilton Kleber Gonçalves Do Patrocinio, Guillaume Gravier, et al.. Hierarchical Multi-Label Propagation using Speaking Face Graphs for Multimodal Person Discovery. Multimedia Tools and Applications, In press, pp.1-27. ⟨10.1007/s11042-020-09692-x⟩. ⟨hal-02926035⟩
127 View
267 Download

Altmetric

Share

More