The characteristics like density of objects, their contrast with respect to surrounding background, their occlu-sion level and many more describe the context of the scene. The variation of the context represents ambiguous task to be solved by tracker. In this paper we present a new long term tracking framework boosted by context around each track-let. The framework works by first learning the database of optimal tracker parameters for various context offline. During the testing, the context surrounding each tracklet is extracted and match against database to select best tracker parameters. The tracker parameters are tuned for each tracklet in the scene to highlight its discrimination with respect to surrounding context rather than tuning the parameters for whole scene. The proposed framework is trained on 9 public video sequences and tested on 3 unseen sets. It outperforms the state-of-art pedestrian trackers in scenarios of motion changes, appearance changes and occlusion of objects.