Self-Supervised Video Representation Learning via Latent Time Navigation - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Self-Supervised Video Representation Learning via Latent Time Navigation

Abstract

Self-supervised video representation learning aimed at maximizing similarity between different temporal segments of one video, in order to enforce feature persistence over time. This leads to loss of pertinent information related to temporal relationships, rendering actions such as `enter' and `leave' to be indistinguishable. To mitigate this limitation, we propose Latent Time Navigation (LTN), a time-parameterized contrastive learning strategy that is streamlined to capture fine-grained motions. Specifically, we maximize the representation similarity between different video segments from one video, while maintaining their representations time-aware along a subspace of the latent representation code including an orthogonal basis to represent temporal changes. Our extensive experimental analysis suggests that learning video representations by LTN consistently improves performance of action classification in fine-grained and human-oriented tasks (e.g., on Toyota Smarthome dataset). In addition, we demonstrate that our proposed model, when pre-trained on Kinetics-400, generalizes well onto the unseen real world video benchmark datasets UCF101 and HMDB51, achieving state-of-the-art performance in action recognition.
Fichier principal
Vignette du fichier
25416-Article Text-29479-1-2-20230626_Self-supervised_video_representaiton_learning_AAAI2023.pdf (1.9 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04236128 , version 1 (10-10-2023)

Identifiers

Cite

Di Yang, Yaohui Wang, Quan Kong, Antitza Dantcheva, Lorenzo Garattoni, et al.. Self-Supervised Video Representation Learning via Latent Time Navigation. AAAI 2023 - AAAI Conference on Artificial Intelligence, Feb 2023, Washigton, D.C., United States. ⟨10.1609/aaai.v37i3.25416⟩. ⟨hal-04236128⟩
20 View
26 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More