In many domains such as health monitoring, the semantic information provided by automatic monitoring systems has become essential. These systems should be as robust, as easy to deploy and as affordable as possible. This paper presents a monitoring system for mid to long-term event recognition based on RGB-D (Red Green Blue + Depth) standard algorithms and on additional algorithms in order to address a real world application. Using a hierarchical modelbased approach, the robustness of this system is evaluated on the recognition of physical tasks (e.g., balance test) undertaken by older people (N = 30) during a clinical protocol devoted to dementia study. The performance of the system is demonstrated at recognizing, first, human postures, and second, complex events based on posture and 3D contextual information of the scene.