This article proposes a flexible tracker which can estimate motion and deformations of 3D objects by considering their appearances as nonrigid surfaces. In this approach, a flexible model is built by matching features (key points) over training sequences and by learning the deformations of a spline based model. This statistical model captures the variations in the appearance of objects caused by 3D pose variations. Visual tracking is then possible, for each new frame, by matching local features of the model according to their local appearances as well as optimal optimization of the constraints provided by the flexible model. The approach is demonstrated on real-world images sequences.