Unlocking Pre-trained Image Backbones for Semantic Image Synthesis - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2024

Unlocking Pre-trained Image Backbones for Semantic Image Synthesis

Abstract

Semantic image synthesis, i.e., generating images from user-provided semantic label maps, is an important conditional image generation task as it allows to control both the content as well as the spatial layout of generated images. Although diffusion models have pushed the state of the art in generative image modeling, the iterative nature of their inference process makes them computationally demanding. Other approaches such as GANs are more efficient as they only need a single feed-forward pass for generation, but the image quality tends to suffer when modeling large and diverse datasets. In this work, we propose a new class of GAN discriminators for semantic image synthesis that generates highly realistic images by exploiting feature backbones pretrained for tasks such as image classification. We also introduce a new generator architecture with better context modeling and using cross-attention to inject noise into latent variables, leading to more diverse generated images. Our model, which we dub DP-SIMS, achieves state-of-the-art results in terms of image quality and consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes, surpassing recent diffusion models while requiring two orders of magnitude less compute for inference.
Fichier principal
Vignette du fichier
Semantic_Synthesis.pdf (6.04 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04381466 , version 1 (09-01-2024)
hal-04381466 , version 2 (05-04-2024)

Licence

Identifiers

Cite

Tariq Berrada, Jakob Verbeek, Camille Couprie, Karteek Alahari. Unlocking Pre-trained Image Backbones for Semantic Image Synthesis. CVPR 2024 - IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2024, Seattle, United States. pp.1-21. ⟨hal-04381466v2⟩
65 View
39 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More