Use of video surveillance has substantially increased in the last few decades. Modern video surveillance systems are equipped with techniques that allow traversal of data in an effective and efficient manner, giving massive powers to operators and potentially compromising the privacy of anyone observed by the system. Several techniques to protect the privacy of individuals have therefore been proposed, but very little research work has focused on the specific security requirements of video surveillance data (in transit or in storage) and on authorizing access to this data. In this paper, we present a general model of video surveillance systems that will help identify the major security and privacy requirements for a video surveillance system and we use this model to identify practical challenges in ensuring the security of video surveillance data in all stages (in transit and at rest). Our study shows a gap between the identified security requirements and the proposed security solutions where future research efforts may focus in this domain.