Parallel transport is a fundamental tool to perform statistics on Riemannian manifolds. Since closed formulae don't exist in general, practitioners often have to resort to numerical schemes. Ladder methods are a popular class of algorithms that rely on iterative constructions of geodesic parallelograms. And yet, the literature lacks a clear analysis of their convergence performance. In this work, we give Taylor approximations of the elementary constructions of Schild's ladder and the pole ladder with respect to the Riemann curvature of the underlying space. We then prove that these methods can be iterated to converge with quadratic speed, even when geodesics are approximated by numerical schemes. We also contribute a new link between Schild's ladder and the Fanning Scheme which explains why the latter naturally converges only linearly. The extra computational cost of ladder methods is thus easily compensated by a drastic reduction of the number of steps needed to achieve the requested accuracy. Illustrations on the 2-sphere, the space of symmetric positive definite matrices and the special Euclidean group show that the theoretical errors we have established are measured with a high accuracy in practice. The special Euclidean group with an anisotropic left-invariant metric is of particular interest as it is a tractable example of a non-symmetric space in general , which reduces to a Riemannian symmetric space in a particular case. As a secondary contribution, we compute the covariant derivative of the curvature in this space.