With the increasing importance of fiber tracking in diffusion tensor images for clinical needs, there has been a growing demand for an objective mathematical framework to perform quantitative analysis of white matter fiber bundles incorporating their underlying physical significance. This paper presents such a novel mathematical framework that facilitates mathematical operations between tracts using an inner product based on Gaussian processes, between fibers which span a metric space. This metric facilitates combination of fiber tracts, rendering operations like tract membership to a bundle or bundle similarity simple. Based on this framework, we have designed an automated unsupervised atlas-based clustering method that does not require manual initialization nor an a priori knowledge of the number of clusters. Quantitative analysis can now be performed on the clustered tract volumes across subjects thereby avoiding the need for point parametrization of these fibers, or the use of medial or envelope representations as in previous work. Experiments on synthetic data demonstrate the mathematical operations. Subsequently, the applicability of the unsupervised clustering framework has been demonstrated on a 21 subject dataset.