Multi-person tracking with a robotic platform is one of the cornerstones of human-robot interaction. Challenges arise from occlusions, appearance changes and a time-varying number of people. Furthermore, the final system is constrained by the hardware platform: low computational capacity and limited field-of-view. In this paper, we propose a novel method to simultaneously track a time-varying number of persons in three-dimension and perform visual servoing. The complementary nature of the tracking and visual servoing enables the system to: (i) track multiple objects while compensating for large ego-movements and (ii) visually-control the robot to keep the person-of-interest in the field-of-view. We implement a variational approximation allowing us to effectively solve the inference problem through the use of closed-form solutions. Importantly, this leads to a computationally light system that runs at 10 FPS. The experiments on the NAO-MPVS dataset confirm the importance of using motor information when tracking multiple persons.