Beyond the Camera: Neural Networks in World Coordinates - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2020

Beyond the Camera: Neural Networks in World Coordinates

Résumé

Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information. This fundamental system has been missing from video understanding with deep networks, typically limited to 224 by 224 pixel content locked to the camera frame. We propose a simple idea, WorldFeatures, where each feature at every layer has a spatial transformation, and the feature map is only transformed as needed. We show that a network built with these WorldFeatures, can be used to model eye movements, such as saccades, fixation, and smooth pursuit, even in a batch setting on pre-recorded video. That is, the network can for example use all 224 by 224 pixels to look at a small detail one moment, and the whole scene the next. We show that typical building blocks, such as convolutions and pooling, can be adapted to support WorldFeatures using available tools. Experiments are presented on the Charades, Olympic Sports, and Caltech-UCSD Birds-200-2011 datasets, exploring action recognition, fine-grained recognition, and video stabilization.
Fichier principal
Vignette du fichier
2003.05614.pdf (7.62 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03990556 , version 1 (15-02-2023)

Licence

Paternité

Identifiants

Citer

Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Karteek Alahari. Beyond the Camera: Neural Networks in World Coordinates. 2020. ⟨hal-03990556⟩
21 Consultations
14 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More