We present a new evaluation methodology to better evaluate video processing performance. Recent evaluation methods depend heavily on the benchmark dataset. The result may be different if we change the testing video sequences. The difference is mainly due to the video sequence content which usually includes many video processing problems (illumination changes, weak contrast etc.) at different difficulty levels. Hence it is difficult to extrapolate the evaluation result on new sequences. In this paper, we propose an evaluation methodology that help to reuse the evaluation result. We try to isolate each video processing problem and define quantitative measures to compute the difficulty level of a video relatively to the given problem. The maximum difficulty level of the videos at which the algorithm is performing good enough is defined as the upper bound of the algorithm capacity for handling the problem. To illustrate this methodology, we present metrics that evaluate the algorithm performance relatively to the problems of handling weakly contrasted objects and shadows.