ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles Computer Graphics Forum Year : 2023

ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes

Abstract

Recent advances in Neural Radiance Fields enable the capture of scenes with motion. However, editing the motion is hard; no existing method allows editing beyond the space of motion existing in the original video, nor editing based on physics. We present the first approach that allows physically-based editing of motion in a scene captured with a single hand-held video camera, containing vibrating or periodic motion. We first introduce a Lagrangian representation, representing motion as the displacement of particles, which is learned while training a radiance field. We use these particles to create a continuous representation of motion over the sequence, which is then used to perform a modal analysis of the motion thanks to a Fourier transform on the particle displacement over time. The resulting extracted modes allow motion synthesis, and easy editing of the motion, while inheriting the ability for free-viewpoint synthesis in the captured 3D scene from the radiance field. We demonstrate our new method on synthetic and real captured scenes.
Fichier principal
Vignette du fichier
modal_nerf_submission.pdf (24.08 Mo) Télécharger le fichier
modal_nerf.mp4 (113.98 Mo) Télécharger le fichier
Vignette du fichier
teaser.jpg (120.65 Ko) Télécharger le fichier
Format Video
Format Figure, Image

Dates and versions

hal-04131503 , version 1 (22-06-2023)

Identifiers

Cite

Automne Petitjean, Yohan Poirier-Ginter, Ayush Tewari, Guillaume Cordonnier, George Drettakis. ModalNeRF: Neural Modal Analysis and Synthesis for Free-Viewpoint Navigation in Dynamically Vibrating Scenes. Computer Graphics Forum, 2023, 42 (4), pp.e14888. ⟨10.1111/cgf.14888⟩. ⟨hal-04131503⟩
436 View
67 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More