In this paper we propose a new method to capture the temporal evolution of a surface from multiple videos. By contrast to most current methods, our algorithm does not use any prior information on t he nature of the tracked surface. In addition, it does not require sparse features to constrain the deformation but only relies on purely geometric information: a target set of 3D points and normals. Our approach is inspired by the Iterative Closest Point algorithm but handles large deformati ons of non-rigid surfaces. To this aim, a mesh is iteratively deformed while enforcing local rigidity with respect to a reference model. This rigidity is preserved by diffusing it on local patches randomly seeded on the surface. The iterative nature of the algorithm combined with the softly enforced local rigidity allows to progressively evolve the mesh to fit the target data. The proposed method is validated and evaluated on several standard and challenging surface data sets acquired using real videos.