Deep Residual Architecture Using Pixel and Feature Cues for View Synthesis and Temporal Interpolation - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue IEEE Transactions on Computational Imaging Année : 2022

Deep Residual Architecture Using Pixel and Feature Cues for View Synthesis and Temporal Interpolation

Résumé

In this paper, we propose a deep residual architecture that can be used both for synthesizing high quality angular views in light fields and temporal frames in classical videos. The proposed framework consists of an optical flow estimator optimized for view synthesis, a trainable feature extractor and a residual convolutional network for pixel and feature-based view reconstruction. Among these modules, the fine-tuning of the optical flow estimator specifically for the view synthesis task yields scene depth or motion information that is well optimized for the targeted problem. In cooperation with the end-to-end trainable encoder, the synthesis block employs both pixel-based and feature-based synthesis with residual connection blocks, and the two synthesized views are fused with the help of a learned soft mask to obtain the final reconstructed view. Experimental results with various datasets show that our method performs favorably against other state-of-the-art (SOTA) methods with a large gain for light field view synthesis. Furthermore, with a little modification, our method can also be used for video frame interpolation, generating high quality frames compared with SOTA interpolation methods.
Fichier principal
Vignette du fichier
TCI_2021.pdf (25.01 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03607417 , version 1 (13-03-2022)

Identifiants

Citer

Jinglei Shi, Xiaoran Jiang, Christine Guillemot. Deep Residual Architecture Using Pixel and Feature Cues for View Synthesis and Temporal Interpolation. IEEE Transactions on Computational Imaging, 2022, pp.1-14. ⟨10.1109/tci.2022.3160671⟩. ⟨hal-03607417⟩
67 Consultations
40 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More