In recent years considerable advances have been made in learning to recognize and localize visual object classes from images annotated with global image-level labels, bounding boxes, or pixel-level segmentations. A second line of research uses unsupervised learning methods such as aspect models to automatically discover the latent object classes of unlabeled image collections. Here we learn spatial aspect models from image-level labels and use them to recover labeled regions in new images. Our models combine low-level texture, color and position cues with spatial random field models that capture the local coherence of region labels. We study two spatial inference models: one based on averaging over forests of minimal spanning trees linking neighboring image regions, the other on an efficient chain-merging Expectation Propagation method for regular 8-neighbor Markov random fields. Experimental results on the MSR Cambridge data sets show that incorporating spatial terms in the aspect model significantly improves the region-level classification rates. So much so, that the spatial random field model trained from image labels only outperforms PLSA trained from segmented images.