Falls in the elderly is a major public health problem because of their frequency and their medical and social consequences. New smart assistive technologies and Health Telematics make it possible to provide elderly with more security and well being at home. A smart home can automatically monitor home activities for early warning in health changes or detecting dangerous situations. One of our objectives is to design an automatic system to detect fall at home, which in its final version will be made up of a network of RGB-D sensors. In this paper, we present a simple and robust method based on the identification and tracking of the center of mass of people evolving in an indoor environment. Using a simple Hidden Markov Model whose observations are the position of the center of mass, its velocity and the general shape of the body, we can surprisingly monitor the activity of a person with high accuracy and thus detect falls with very good accuracy without false positives. An experimental study, that is reported here, has been driven in our smart apartment lab. 26 subjects were asked to perform a predefined scenario in which they realized a set of eight postures. 2 hours of video (216 000 frames) were recorded for the evaluation, half of it being used for the training of the model. The system detected the falls without false positives. This result encourages us to use this system in real situation for a better study of its efficiency.