In this paper we present a method to iteratively capture the dynamic evolution of a surface from a set of point clouds independently acquired from multi-view videos. This is done without prior knowledge on the observed shape and simply deforms the first reconstructed mesh across the sequence to fit these point clouds while preserving the local rigidity with respect to this reference pose. The deformation of this mesh is guided by control points that are randomly seeded on the surface, and around which rigid motions are locally averaged. These rigid motions are computed by iteratively re-establishing point-to-point associations between the deformed mesh and the target data in a way inspired by ICP. Our method introduces a way to account for the point dynamics when establishing these correspondences, a higher level rigidity model between the control points and a coarse-to-fine strategy that allows to fit the temporally inconsistent data more precisely. Experimental results, including quantitative analysis, on standard and challenging datasets obtained from real video sequences show the robustness and the precision of the proposed scheme.