During the last decade, the emerging technology for video retrieval is mainly based on the content. However, semantic-based video retrieval has become more and more necessary for the humans especially the naive users who can only use the human language during retrieval. In this paper, we focus on semantic representation using topological and directional relations between non-moving and moving objects for security using CCTV(closed-circuit Television). In this paper, we propose new spatio-temporal relation to extend previous work using topological and directional relations and investigate spatio-temporal predicates which propose our models. In the experiment part, we compared retrieval results using TSR(Tangent Space Representation) with those using rules represented by the proposed model.