This work studies the quantitative stability of the quadratic optimal transport map between a fixed probability density ρ and a probability measure µ on R^d , which we denote Tµ. Assuming that the source density ρ is bounded from above and below on a compact convex set, we prove that the map µ → Tµ is bi-Hölder continuous on large families of probability measures, such as the set of probability measures whose moment of order p > d is bounded by some constant. These stability estimates show that the linearized optimal transport metric W2,ρ(µ, ν) = Tµ − Tν L 2 (ρ,R d) is bi-Hölder equivalent to the 2-Wasserstein distance on such sets, justifiying its use in applications.