No Reason for No Supervision: Improved Generalization in Supervised Models - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

No Reason for No Supervision: Improved Generalization in Supervised Models


We consider the problem of training a deep neural network on a given classification task, e.g., ImageNet-1K (IN1K), so that it excels at both the training task as well as at other (future) transfer tasks. These two seemingly contradictory properties impose a trade-off between improving the model's generalization and maintaining its performance on the original task. Models trained with self-supervised learning tend to generalize better than their supervised counterparts for transfer learning; yet, they still lag behind supervised models on IN1K. In this paper, we propose a supervised learning setup that leverages the best of both worlds. We extensively analyze supervised training using multi scale crops for data augmentation and an expendable projector head, and reveal that the design of the projector allows us to control the trade off between performance on the training task and transferability. We further replace the last layer of class weights with class prototypes computed on the fly using a memory bank and derive two models: t-ReX that achieves a new state of the art for transfer learning and outperforms top methods such as DINO and PAWS on IN1K, and t-ReX* that matches the highly optimized RSB-A1 model on IN1K while performing better on transfer tasks. Code and pretrained models:
Fichier principal
Vignette du fichier
t_Rex_ICLR_2023.pdf (5.2 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03929621 , version 1 (08-01-2023)
hal-03929621 , version 2 (10-03-2023)



Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, Diane Larlus. No Reason for No Supervision: Improved Generalization in Supervised Models. ICLR 2023 - International Conference on Learning Representations, May 2023, Kigali, Rwanda. pp.1-26. ⟨hal-03929621v2⟩
297 View
189 Download



Gmail Facebook X LinkedIn More