This paper introduces a new image representation relying on the spatial pooling of geometrically consistent visual matches. We therefore introduce a new match kernel based on the in- verse rank of the shared nearest neighbors combined with local geometric constraints. To avoid overtting and reduce processing costs, the dimensionality of the resulting over- complete representation is further reduced by hierarchically pooling the raw consistent matches according to their spa- tial position in the training images. The nal image repre- sentation is obtained by concatenating the resulting feature vectors at several resolutions. Learning from these represen- tations using a logistic regression classier is shown to pro- vide excellent ne-grained classication performances out- performing the results reported in the literature on several classication tasks.