A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles IEEE MultiMedia Year : 2018

A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking


With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled data, unsupervised learning algorithms became very popular for organizing and retrieving large video collections in a task defined as video hyperlinking. Information stored as videos typically contain two modalities, namely an audio and a visual one, that are used conjointly in multimodal systems by undergoing fusion. Multimodal autoencoders have been long used for performing multimodal fusion. In this work, we start by evaluating different initial, single-modal representations for automatic speech transcripts and for video keyframes. We progress to evaluating different autoencoding methods of performing multimodal fusion in an offline setup. The best performing setup is then evaluated in a live setup at TRECVID's 2016 video hyperlinking task. As in offline evaluations, we show that focusing on crossmodal translations as a way of performing multimodal fusion yields improved multimodal representations and that our simple system, trained in an unsupervised manner, with no external information information, defines the new state of the art in a live video hyperlinking setup. We conclude by performing an analysis on data gathered after the live evaluations at TRECVID 2016 and express our thoughts on the overall performance of our proposed system.
Fichier principal
Vignette du fichier
Vukotic - A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking - last revision.pdf (4.19 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01848539 , version 1 (24-07-2018)
hal-01848539 , version 2 (20-08-2018)



Vedran Vukotić, Christian Raymond, Guillaume Gravier. A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking. IEEE MultiMedia, 2018, 25 (2), pp.11-23. ⟨10.1109/MMUL.2018.023121161⟩. ⟨hal-01848539v2⟩
410 View
456 Download



Gmail Facebook X LinkedIn More