Airbert: In-domain Pretraining for Vision-and-Language Navigation - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2021

Airbert: In-domain Pretraining for Vision-and-Language Navigation


Vision-and-language navigation (VLN) aims to enable embodied agents to navigate in realistic environments using natural language instructions. Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging. Recent methods explore pretraining to improve generalization, however, the use of generic image-caption datasets or existing smallscale VLN environments is suboptimal and results in limited improvements. In this work, we introduce BnB 1 , a largescale and diverse in-domain VLN dataset. We first collect image-caption (IC) pairs from hundreds of thousands of listings from online rental marketplaces. Using IC pairs we next propose automatic strategies to generate millions of VLN path-instruction (PI) pairs. We further propose a shuffling loss that improves the learning of temporal order inside PI pairs. We use BnB to pretrain our Airbert 2 model that can be adapted to discriminative and generative settings and show that it outperforms state of the art for Room-to-Room (R2R) navigation and Remote Referring Expression (REVERIE) benchmarks. Moreover, our in-domain pretraining significantly increases performance on a challenging few-shot VLN evaluation, where we train the model only on VLN instructions from a few houses.
Fichier principal
Vignette du fichier
2108.09105.pdf (43.12 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03470013 , version 1 (08-12-2021)


  • HAL Id : hal-03470013 , version 1


Pierre-Louis Guhur, Makarand Tapaswi, Shizhe Chen, Ivan Laptev, Cordelia Schmid. Airbert: In-domain Pretraining for Vision-and-Language Navigation. ICCV 2021 - International Conference on Computer Vision, Oct 2021, Virtual, France. ⟨hal-03470013⟩
49 View
7 Download


Gmail Facebook X LinkedIn More