Learning Video-Conditioned Policies for Unseen Manipulation Tasks - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2023

Learning Video-Conditioned Policies for Unseen Manipulation Tasks


The ability to specify robot commands by a non-expert user is critical for building generalist agents capable of solving a large variety of tasks. One convenient way to specify the intended robot goal is by a video of a person demonstrating the target task. While prior work typically aims to imitate human demonstrations performed in robot environments, here we focus on a more realistic and challenging setup with demonstrations recorded in natural and diverse human environments. We propose Video-conditioned Policy learning (ViP), a data-driven approach that maps human demonstrations of previously unseen tasks to robot manipulation skills. To this end, we learn our policy to generate appropriate actions given current scene observations and a video of the target task. To encourage generalization to new tasks, we avoid particular tasks during training and learn our policy from unlabelled robot trajectories and corresponding robot videos. Both robot and human videos in our framework are represented by video embeddings pre-trained for human action recognition. At test time we first translate human videos to robot videos in the common video embedding space, and then use resulting embeddings to condition our policies. Notably, our approach enables robot control by human demonstrations in a zero-shot manner, i.e., without using robot trajectories paired with human instructions during training. We validate our approach on a set of challenging multi-task robot manipulation environments and outperform state of the art. Our method also demonstrates excellent performance in a new challenging zero-shot setup where no paired data is used during training.
Fichier principal
Vignette du fichier
2305.06289.pdf (1.17 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04226856 , version 1 (03-10-2023)




Elliot Chane-Sane, Cordelia Schmid, Ivan Laptev. Learning Video-Conditioned Policies for Unseen Manipulation Tasks. 2023 IEEE International Conference on Robotics and Automation (ICRA), May 2023, London, United Kingdom. pp.909-916, ⟨10.1109/ICRA48891.2023.10161336⟩. ⟨hal-04226856⟩
21 View
39 Download



Gmail Mastodon Facebook X LinkedIn More