Beyond the Camera: Neural Networks in World Coordinates - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Preprints, Working Papers, ... Year :

Beyond the Camera: Neural Networks in World Coordinates


Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information. This fundamental system has been missing from video understanding with deep networks, typically limited to 224 by 224 pixel content locked to the camera frame. We propose a simple idea, WorldFeatures, where each feature at every layer has a spatial transformation, and the feature map is only transformed as needed. We show that a network built with these WorldFeatures, can be used to model eye movements, such as saccades, fixation, and smooth pursuit, even in a batch setting on pre-recorded video. That is, the network can for example use all 224 by 224 pixels to look at a small detail one moment, and the whole scene the next. We show that typical building blocks, such as convolutions and pooling, can be adapted to support WorldFeatures using available tools. Experiments are presented on the Charades, Olympic Sports, and Caltech-UCSD Birds-200-2011 datasets, exploring action recognition, fine-grained recognition, and video stabilization.
Fichier principal
Vignette du fichier
2003.05614.pdf (7.62 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03990556 , version 1 (15-02-2023)



Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Karteek Alahari. Beyond the Camera: Neural Networks in World Coordinates. 2020. ⟨hal-03990556⟩
5 View
1 Download



Gmail Facebook Twitter LinkedIn More