Multimodal Vision Transformers with Forced Attention for Behavior Analysis - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Multimodal Vision Transformers with Forced Attention for Behavior Analysis


Human behavior understanding requires looking at minute details in the large context of a scene containing multiple input modalities. It is necessary as it allows the design of more human-like machines. While transformer approaches have shown great improvements, they face multiple challenges such as lack of data or background noise. To tackle these, we introduce the Forced Attention (FAt) Transformer which utilize forced attention with a modified backbone for input encoding and a use of additional inputs. In addition to improving the performance on different tasks and inputs, the modification requires less time and memory resources. We provide a model for a generalised feature extraction for tasks concerning social signals and behavior analysis. Our focus is on understanding behavior in videos where people are interacting with each other or talking into the camera which simulates the first person point of view in social interaction. FAt Transformers are applied to two downstream tasks: personality recognition and body language recognition. We achieve state-of-the-art results for Udiva v0.5, First Impressions v2 and MPII Group Interaction datasets. We further provide an extensive ablation study of the proposed architecture.
Fichier principal
Vignette du fichier
WACV_2023__Multimodal_Vision_Transformers_with_Forced_Attention_for_Behavior_Analysis.pdf (409.67 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Public Domain

Dates and versions

hal-03936484 , version 1 (12-01-2023)


Public Domain



Tanay Agrawal, Michal Balazia, Philipp Müller, Francois F Bremond. Multimodal Vision Transformers with Forced Attention for Behavior Analysis. WACV '23: IEEE International Winter Conference on Applications in Computer Vision, IEEE, Jan 2023, Waikoloa, United States. ⟨hal-03936484⟩
44 View
63 Download



Gmail Mastodon Facebook X LinkedIn More