The paper contributes to presenting both an accurate and robust shape tracking algorithm and a novel visual servoing method. Two steps are involved in the tracking algorithm. Firstly the object shape is assumed to vary under an affine model, and the edge detection is performed along the normal lines to the contour. As a result it is possible to use a Kalman filter to perform efficient tracking. The second step concerns image matching based on perspective model, which is achieved iteratively by searching locally along the normal lines also. As to visual servoing we propose to control the translations of the robot with the normalized zeroth and first order image moments, and to control the orientation with rotation axis and angle extracted from a Homography matrix. Two experiments demonstrate that the tracking algorithm is accurate and robust enough to be used in visual servoing, and the novel visual servoing method is superior to traditional ones.