Language Conditioned Spatial Relation Reasoning for 3D Object Grounding - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Language Conditioned Spatial Relation Reasoning for 3D Object Grounding


Localizing objects in 3D scenes based on natural language requires understanding and reasoning about spatial relations. In particular, it is often crucial to distinguish similar objects referred by the text, such as "the left most chair" and "a chair next to the window". In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations. To this end, we design a spatial self-attention layer that accounts for relative distances and orientations between objects in input 3D point clouds. Training such a layer with visual and language inputs enables to disambiguate spatial relations and to localize objects referred by the text. To facilitate the cross-modal learning of relations, we further propose a teacher-student approach where the teacher model is first trained using ground-truth object labels, and then helps to train a student model using point cloud inputs. We perform ablation studies showing advantages of our approach. We also demonstrate our model to significantly outperform the state of the art on the challenging Nr3D, Sr3D and ScanRefer 3D object grounding datasets.
Fichier principal
Vignette du fichier
NeurIPS22_3DVG.pdf (5.37 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03890174 , version 1 (08-12-2022)


  • HAL Id : hal-03890174 , version 1


Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, Ivan Laptev. Language Conditioned Spatial Relation Reasoning for 3D Object Grounding. NeurIPS 2022 - 36th Conference on Neural Information Processing Systems, Nov 2022, New Orleans, United States. ⟨hal-03890174⟩
24 View
16 Download


Gmail Mastodon Facebook X LinkedIn More