A number of Computer Vision and Artificial Intelligence applications are based on descriptors that are extracted from imaged objects. One widely used class of such descriptors are the invariant moments, with Zernike moments being reported as some of the most efficient descriptors. The calculation of image moments requires the definition of distance and angle of any pixel from the centroid pixel. While this is straightforward in images acquired by projective cameras, it is complicated and time consuming for omni-directional images obtained by fish-eye cameras. In this work, we provide an efficient way of calculating moment invariants in time domain from omni-directional images, using the calibration of the acquiring camera. The proposed implementation of the descriptors is assessed in the case of indoor video in terms of classification accuracy of the segmented human silhouettes. Numerical results are presented for different poses of human silhouettes and comparisons between the traditional and the proposed implementation of the Zernike moments are presented. The computational complexity for the proposed implementation is also provided.