This paper proposes a framework to recognize and classify loosely constrained activities with minimal supervision. The framework use basic trajectory information as input and goes up to video interpretation. The work reduces the gap between low-level information and semantic interpretation, building an intermediate layer composed Primitive Events. The proposed representation for primitive events aims at capturing small meaningful motions over the scene with the advantage of been learnt in an unsupervised manner. We propose the modelling of an activity using Primitive Events as the main descriptors. The activity model is built in a semi-supervised way using only real tracking data. Finally we validate the descriptors by recognizing and labelling modelled activities in a home-care application dataset.