Evaluating the quality of tracking outputs is an important task in video analysis. This paper presents a new framework for estimating both detection and tracking quality during runtime. If anomalies are detected in the tracking output results, they are categorized as natural phenomena or real errors using contextual information. As this framework should be generic and work on any kind of system (single camera, camera network), a re-acquisition step using a constrained clustering algorithm is also performed in order to keep track of the object even if it leaves the scene and comes back or appears on another camera. The framework is evaluated on two datasets using different kinds of tracking algorithms.