XCiT: Cross-Covariance Image Transformers - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

XCiT: Cross-Covariance Image Transformers


Following tremendous success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens, i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and highresolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images. Our cross-covariance image transformer (XCiT)-built upon XCA-combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including (self-supervised) image classification on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k. We replace the self-attention, as originally introduced by Vaswani et al. [68], with a "transposed" attention that we denote as "cross-covariance attention" (XCA). Cross-covariance attention substi
Fichier principal
Vignette du fichier
xcit_haltools.pdf (3.35 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03572703 , version 1 (14-02-2022)




  • HAL Id : hal-03572703 , version 1


Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, et al.. XCiT: Cross-Covariance Image Transformers. NeurIPS 2021 - 35th Conference on Neural Information Processing Systems, Dec 2021, Sydney, Australia. pp.1-21. ⟨hal-03572703⟩
151 View
89 Download


Gmail Facebook X LinkedIn More