Learning to Manipulate Tools by Aligning Simulation to Video Demonstration - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Journal Articles IEEE Robotics and Automation Letters Year : 2022

Learning to Manipulate Tools by Aligning Simulation to Video Demonstration


A seamless integration of robots into human environments requires robots to learn how to use existing human tools. Current approaches for learning tool manipulation skills mostly rely on expert demonstrations provided in the target robot environment, for example, by manually guiding the robot manipulator or by teleoperation. In this work, we introduce an automated approach that replaces an expert demonstration with a Youtube video for learning a tool manipulation strategy. The main contributions are twofold. First, we design an alignment procedure that aligns the simulated environment with the realworld scene observed in the video. This is formulated as an optimization problem that finds a spatial alignment of the tool trajectory to maximize the sparse goal reward given by the environment. Second, we devise an imitation learning approach that focuses on the trajectory of the tool and how it interacts with the environment, rather than the motion of the human. We demonstrate the proposed approach on spade, scythe and hammer tools in simulation, and show the effectiveness of the trained policy for the spade on a real Franka Emika Panda robot demonstration.
Fichier principal
Vignette du fichier
Learning to Manipulate Tools byAligning Simulation to Video Demonstration.pdf (8.42 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03478117 , version 1 (13-12-2021)



Kateryna Zorina, Justin Carpentier, Josef Sivic, Vladimír Petrík. Learning to Manipulate Tools by Aligning Simulation to Video Demonstration. IEEE Robotics and Automation Letters, In press, ⟨10.48550/arXiv.2111.03088⟩. ⟨hal-03478117⟩
86 View
58 Download



Gmail Facebook X LinkedIn More