We investigate the use of two visual descriptors: Local Bi-nary Patterns-Three Orthogonal Planes(LBP-TOP) and Dense Trajectories for depression assessment on the AVEC 2014 challenge dataset. We encode the visual information gen-erated by the two descriptors using Fisher Vector encod-ing which has been shown to be one of the best performing methods to encode visual data for image classification. We also incorporate audio features in the final system to intro-duce multiple input modalities. The results produced using Linear Support Vector regression outperform the baseline method[16].