Learning from Unlabeled 3D Environments for Vision-and-Language Navigation - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Learning from Unlabeled 3D Environments for Vision-and-Language Navigation


In vision-and-language navigation (VLN), an embodied agent is required to navigate in realistic 3D environments following natural language instructions. One major bottleneck for existing VLN approaches is the lack of sufficient training data, resulting in unsatisfactory generalization to unseen environments. While VLN data is typically collected manually, such an approach is expensive and prevents scalability. In this work, we address the data scarcity issue by proposing to automatically create a large-scale VLN dataset from 900 unlabeled 3D buildings from HM3D [45]. We generate a navigation graph for each building and transfer object predictions from 2D to generate pseudo 3D object labels by cross-view consistency. We then fine-tune a pretrained language model using pseudo object labels as prompts to alleviate the cross-modal gap in instruction generation. Our resulting HM3D-AutoVLN dataset is an order of magnitude larger than existing VLN datasets in terms of navigation environments and instructions. We experimentally demonstrate that HM3D-AutoVLN significantly increases the generalization ability of resulting VLN models. On the SPL metric, our approach improves over state of the art by 7.1% and 8.1% on the unseen validation splits of REVERIE and SOON datasets respectively.
Fichier principal
Vignette du fichier
ECCV22_3D_VLN_camera_ready.pdf (11.33 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03890196 , version 1 (08-12-2022)


  • HAL Id : hal-03890196 , version 1


Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, Ivan Laptev. Learning from Unlabeled 3D Environments for Vision-and-Language Navigation. ECCV 2022 - European Conference on Computer Vision, Oct 2022, Tel Aviv-Jaffa, Israel. ⟨hal-03890196⟩
42 View
123 Download


Gmail Mastodon Facebook X LinkedIn More