On the duality between contrastive and non-contrastive self-supervised learning - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

On the duality between contrastive and non-contrastive self-supervised learning

Abstract

Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and covariance based non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show the influence (or lack thereof) of design choices on downstream performance. Motivated by our equivalence result, we investigate the low performance of SimCLR and show how it can match VICReg's with careful hyperparameter tuning, improving significantly over known baselines. We also challenge the popular assumption that non-contrastive methods need large output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and non-contrastive methods in certain regimes can be closed given better network design choices and hyperparameter tuning. The evidence shows that unifying different SOTA methods is an important direction to build a better understanding of self-supervised learning.
Fichier principal
Vignette du fichier
main.pdf (674.02 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03685169 , version 1 (02-06-2022)
hal-03685169 , version 2 (02-10-2022)
hal-03685169 , version 3 (19-06-2023)

Licence

Identifiers

Cite

Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. ICLR 2023 - Eleventh International Conference on Learning Representations, May 2023, Kigali, Rwanda. ⟨10.48550/arXiv.2206.02574⟩. ⟨hal-03685169v3⟩
308 View
1155 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More