MARS: Motion-Augmented RGB Stream for Action Recognition - Inria - Institut national de recherche en sciences et technologies du numérique
Communication Dans Un Congrès Année : 2019

MARS: Motion-Augmented RGB Stream for Action Recognition

Résumé

Most state-of-the-art methods for action recognition consist of a two-stream architecture with 3D convolutions: an appearance stream for RGB frames and a motion stream for optical flow frames. Although combining flow with RGB improves the performance, the cost of computing accurate optical flow is high, and increases action recognition latency. This limits the usage of two-stream approaches in real-world applications requiring low latency. In this paper, we introduce two learning approaches to train a standard 3D CNN, operating on RGB frames, that mimics the motion stream, and as a result avoids flow computation at test time. First, by minimizing a feature-based loss compared to the Flow stream, we show that the network reproduces the motion stream with high fidelity. Second, to leverage both appearance and motion information effectively, we train with a linear combination of the feature-based loss and the standard cross-entropy loss for action recognition. We denote the stream trained using this combined loss as Motion- Augmented RGB Stream (MARS). As a single stream, MARS performs better than RGB or Flow alone, for instance with 72.7% accuracy on Kinetics compared to 72.0% and 65.6% with RGB and Flow streams respectively.
Fichier principal
Vignette du fichier
MARS_paper.pdf (522.24 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02140558 , version 1 (27-05-2019)

Identifiants

Citer

Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, Cordelia Schmid. MARS: Motion-Augmented RGB Stream for Action Recognition. CVPR 2019 - IEEE Conference on Computer Vision & Pattern Recognition, Jun 2019, Long Beach, CA, United States. pp.7874-7883, ⟨10.1109/CVPR.2019.00807⟩. ⟨hal-02140558⟩
576 Consultations
2723 Téléchargements

Altmetric

Partager

More