This work deals with the challenge of semantic segmentation based on deep learning methods in the case of realistic scarce ground truth maps. Exhaustive ground truths usually found in benchmark datasets can be used to train deep learning architectures successfully. On the contrary, real-world ground truths are almost never exhaustive, they are spatially sparse and typically they don't represent the spatial borders among the classes, affecting the accuracy of the resulting segmentation maps significantly. Here, the proposed approach addresses precisely this challenge with a novel combination of hierarchical probabilistic graphical models (PGMs) and deep neural networks. The rationale is to exploit the spatial modeling capabilities of hierarchical PGMs to mitigate the impact of incomplete ground truth and obtain accurate classification results in scenarios where exhaustive ground truth does not exist.