The main aim of this article is the description of an original approach which allows the efficient tracking of two-dimensional (2D) patterns in image sequences. This approach consists of two stages. An off-line is devoted to the computation of an interaction matrix linking the aspect variation of a pattern with its displacement in the image. In a second stage, this matrix is used to track the pattern by the following process. At first, the position, the scale and the orientation of the pattern is predicted. Then, the difference between the pattern observed at the predicted position and the reference pattern (which is to be tracked) is computed. Finally, this difference multiplied by the interaction matrix gives a correction to apply to the prediction. Computing this correction has a very low computational cost, allowing a real time implementation of the algorithm. From these principles, we have developed a real time three-dimensional object tracker; objects are modeled by sets of 2D appearances. We show that these algorithms are resistant to occlusions and changes of illumination.