To close the loop between motion and vision, tracked visual features must remain in the camera field of view (visibility constraint). To overcome the visibility constraint , visual servoing methods can benefit from panoramic sensors such as catadioptric cameras (combining both mirrors and lenses). In this paper, we present a vision-based framework to control a nonholonomic mobile robot using a catadioptric imaging system. We particularly focus on a suitable catadioptric image-based control strategy of a nonholonomic robot in order to follow a 3D straight line. Such strategy can be applied to navigate in indoor or urban environment since the extraction and the tracking of straight lines are natural. First the control objectives are formulated in the catadioptric image space. The control law is then designed according to a well suited chained system for a mobile robot state vector directly expressed in the image space using a generic camera model. Simulation results illustrate the control strategy in the case of hypercatadioptric and paracatadioptric cameras. Index Terms-Omnidirectional vision, mobile robot control , visual servoing.