In most neuroimaging studies, one builds a brain template that serves as a reference anatomy for normalizing the measurements of each individual subject into a common space. Such a template should be representative of the population under study, in order to avoid biasing the subsequent statistical analyses. The template is often computed by iteratively registering all images to the current template and then averaging the intensities of the registered images. Geometrically, the procedure can be summarized as follows: it is the computation of the template as the " Fréchet mean " of the images projected in a quotient space. It has been argued recently that this type of algorithm could actually be asymptotically biased, therefore inconsistent. In other words, even with an infinite number of brain images in the database, the template estimate may not converge to the brain anatomy it is meant to estimate. Our paper investigates this phenomenon. We present a methodology that quantifies spatially the brain template's asymptotic bias. We identify the main variables controlling the inconsistency. This leads us to investigate the topology of the template's intensity levels sets, represented by its Morse-Smale complex. We propose a topologically constrained adaptation of the template computation, that constructs a hierarchical template with bounded bias. We apply our method to the analysis of a brain template of 136 T1 weighted MR images from the Open Access Series of Imaging Studies (OASIS) database.