In this paper we present a geometrically-constrained time delay estimation method for sound source localization (gTDE). An algebraic analysis reveals that the method can deal with an arbitrary number of non-coplanar microphones. We derive a constrained non-linear optimization problem that can be solved using local convex programming. Unlike existing techniques, which consider pairwise TDE's, the proposed method optimally estimates a set of time delays that are consistent with the source's location. Extensive simulated experiments validate the method in the presence of noise and of reverberations.