Medical imaging protocols produce large amounts of multi- modal volumetric images. The large size of the datasets contributes to the success of supervised discriminative methods for semantic image segmentation. Classifying relevant structures in medical images is challenging due to (a) the large size of data volumes, and (b) the severe class overlap in the feature space. Subsampling the training data addresses the first issue at the cost of discarding potentially useful image information. Increasing feature dimensionality addresses the second but requires dense sampling. We propose a general and efficient solution to these problems. "Spatially Adaptive Random Forests" (SARF) is a supervised learning algorithm. SARF aims at automatic semantic labelling of large medical volumes. During training, it learns the optimal image sampling associated to the classification task. During testing, the algorithm quickly handles the background and focuses challenging image regions to refine the classification. SARF demonstrated top performance in the context of multi-class gliomas segmentation in multi-modal MR images.