This paper presents a novel and robust Markov random field (MRF) approach for matching sparse regions between two images related by a non-rigid transformation. The proposed MRF model incorporates appearance based region similarities as well as the spatial correlations of neighboring regions. The method involves initial affine covariant region detection in each image individually followed by a viewpoint invariant representation of the detected regions using region descriptors. The task of matching these descriptors is performed by estimating the maximum a posteriori (MAP) labeling of the proposed MRF model using Belief Propagation. In particular, we introduce a new pair-wise geometric constraint, which is evaluated on the texture of the detected regions while taking the local image geometry into account. The evaluation of the geometric and appearance constraints in the same space allows for their combination in the MRF objective function without a need for learning or tuning any weighting parameters. The performance of the method is compared to that of the current state-of-the-art matching algorithms using datasets with varying levels of difficulties.