Class hierarchies are commonly used to reduce the complexity of the classification problem. This is crucial in situations when one has to deal with multiple categories. In this work, we evaluate the suitability of class hierarchies currently constructed for visual recognition. We show that top-down as well as bottom-up approaches that are commonly used to automatically construct hierarchies, incorporate assumptions about separability of classes that cannot be fulfilled in the case of visual recognition of a large number of object categories. We propose a modification which is appropriate for most top-down approaches. It allows to construct better class hierarchies that postpone decisions in the presence of uncertainty and thus provide higher recognition accuracy. We also compare our method to flat one-against-all approach and show how to control the speed-for-accuracy trade-off by using our method. For the experimental evaluation, we use the Caltech-256 visual object classes dataset and compare to the state-of-the-art.