Image-based deformations are generally used for visual tracking of deformable objects moving in the 3D space. For the visual tracking of deformable objects, this assumption has shown to give good results. However it is not satisfying for the visual tracking of 3D rigid objects as the underlying structure cannot be directly estimated. The general belief is that obtaining the 3D structure directly is difficult. In this article, we propose a parameterization that is well adapted either to track deformable objects or to recover the structure of 3D objects. Furthermore, the formulation leads to an efficient implementation that can considerably reduce the computational load and it is therefore more adapted to real-time robotic applications. Experiments with simulated and real data validate the approach for deformable object visual tracking and 3D structure estimation. The computational efficiency is also compared to standard methods.