We propose an activity monitoring framework based on a platform called VSIP, enabling behavior recognition in different environments. To allow end-users to actively participate in the development of a new application, VSIP separates algorithms from a priori knowledge. For describing how VSIP works, we present a full description of a system developed with this platform for recognizing behaviors, involving either isolated individual, group of people or crowds, in the context of visual monitoring of metro scenes using multiple cameras. In this work, we also illustrate the capability of the framework to easily combine and tune various recognition methods dedicated to the visual analysis of specific situations (e.g. mono/multi actors activities, numerical/symbolic actions or temporal scenarios). We also present other applications using this framework, in the context of behavior recognition. VSIP has shown a good performance on human behavior recognition for different problems and configurations, being suitable to fulfill a large variety of requirements.