This paper examines a novel approach for temporal calibration of a 3D freehand ultrasound (US) system. A localization system fixed on the probe gives the position and orientation of the probe. For quantitative use, calibration is needed to correctly localize a B-scan in 4D (3D+t) space. Temporal latency estimation is defined in a general robust formulation using no specific probe motion constraints. Experiments were performed on synthetic and real data using a 3D freehand ultrasound system. The achieved precision is lower than the image acquisition rate ($40ms$). A validation study using a calibration phantom has been performed to evaluate the influence of incorrect latency estimation on the 3D reconstruction procedure. We showed that for latency estimation errors less than $40ms$, the 3D reconstruction errors are negligible for volume estimation.