We introduce in this paper a novel structure-enhanced VP detector, CollabVP, which uses a multi-task learning framework to estimate multiple horizontal vanishing points (VPs) from a single RGB image. CollabVP exploits contextual information and scene structures through masks of vertical structures to accurately estimate the horizon line and VP positions. The proposed approach is not limited to Manhattan worlds and can detect any number of VPs.