Characterizing an image region by its feature inter-correlations is a modern trend in computer vision. In this paper, we introduce a new image descriptor that can be seen as a natural extension of a covariance descriptor with the advantage of capturing nonlinear and non-monotone dependencies. Inspired from the recent advances in mathematical statistics of Brownian motion, we can express highly complex structural information in a compact and computationally efficient manner. We show that our Brownian covariance descriptor can capture richer image characteristics than the covariance descriptor. Additionally, a detailed analysis of the Brownian manifold reveals that in opposite to the classical covariance descriptor, the proposed descriptor lies in a relatively flat manifold, which can be treated as a Euclidean. This brings significant boost in the efficiency of the descriptor. The effectiveness and the generality of our approach is validated on two challenging vision tasks, pedestrian classification and person re-identification. The experiments are carried out on multiple datasets achieving promising results.