Reconstructing and grounding narrated instructional videos in 3D - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Preprints, Working Papers, ... Year : 2021

Reconstructing and grounding narrated instructional videos in 3D


Narrated instructional videos often show and describe manipulations of similar objects, e.g., repairing a particular model of a car or laptop. In this work we aim to reconstruct such objects and to localize associated narrations in 3D. Contrary to the standard scenario of instancelevel 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product. Narrations may also have large variation in natural language expressions. We address these challenges by three contributions. First, we propose an approach for correspondence estimation combining learnt local features and dense flow. Second, we design a two-step divide and conquer reconstruction approach where the initial 3D reconstructions of individual videos are combined into a 3D alignment graph. Finally, we propose an unsupervised approach to ground natural language in obtained 3D reconstructions. We demonstrate the effectiveness of our approach for the domain of car maintenance. Given raw instructional videos and no manual supervision, our method successfully reconstructs engines of different car models and associates textual descriptions with corresponding objects in 3D.
Fichier principal
Vignette du fichier
paper.pdf (3.74 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03571657 , version 1 (14-02-2022)


  • HAL Id : hal-03571657 , version 1


Dimitri Zhukov, Ignacio Rocco, Ivan Laptev, Josef Sivic, Johannes L Schönberger, et al.. Reconstructing and grounding narrated instructional videos in 3D. 2021. ⟨hal-03571657⟩
46 View
30 Download


Gmail Facebook X LinkedIn More