People detection on static images and video sequences is a critical task in many computer vision applications, like image retrieval and video surveillance. It is also one of most challenging task due to the large number of possible situations, including variations in people appearance and poses. The proposed approach optimizes an existing approach based on classification on Riemannian manifolds using covariance matrices in a boosting scheme, making training and detection faster while maintaining equivalent performances. This optimisation is achieved by clustering negative samples before training, providing a smaller number of cascade levels and less weak classifiers in most levels in comparison with the original approach. Our work was evaluated and validated on INRIA Person dataset.