This paper presents an image-based visual servo- ing for controlling the position and orientation of a quadrotor using a fixed downward camera observing landmarks on the level ground. In the proposed method, the negative feedback to the image moments is used to control the vertical motion and rotation around the roll axis. On the other hand, the negative feedback cannot be used to control the horizontal motion due to under-actuation of a quadrotor. Thus, a novel control method is introduced to control the horizontal motion. Simulations are presented to validate the proposed method.