This paper presents a controller for background subtraction algorithms to detect mobile objects in videos. The controller has two main tasks. The first task is to guide the background subtraction algorithm to update its background representation. To realize this task, the controller has to solve two important problems: removing ghosts (background regions misclassified as object of interest) and managing stationary objects. The controller detects ghosts based on object borders. To manage stationary objects, the controller cooperates with the tracking task to detect faster stationary objects without storing various background layers which are difficult to maintain. The second task is to initialize the parameter values of background subtraction algorithms to adapt to the current conditions of the scene. These parameter values enable the background subtraction algorithms to be as much sensitive as possible and to be consistent with the feedback of classification and tracking task.