The visual focus of attention (VFOA) has been recognized as a prominent conversational cue. We are interested in the VFOA tracking of a group of people involved in social interaction. We note that in this case the participants look either at each other or at an object of interest; therefore they don't always face a camera and, consequently, their gazes (and their VFOAs) cannot be based on eye detection and tracking. We propose a method that exploits the correlation between gaze direction and head orientation. Both VFOA and gaze are modeled as latent variables in a Bayesian switching linear dynamic model. The proposed formulation leads to a tractable learning procedure and to an efficient gaze-and-VFOA tracking algorithm. The method is tested and benchmarked using a publicly available dataset that contains typical multi-party human-robot interaction scenarios, and that was recorded with both a motion capture system, and with a camera mounted onto a robot head.