Approximating a matrix by a product of few sparse factors whose supports possess the butterfly structure, which is common to many fast transforms, is key to learn fast transforms and speedup algorithms for inverse problems. We introduce a hierarchical approach, that recursively approximates the considered matrix into two factors. Using recent advances on the well-posedness and tractability of the two-factor fixedsupport sparse matrix factorization problem, we establish exact recovery guarantees for the proposed algorithm. Experiments show that the speed and accuracy of the factorization can be jointly improved by several orders of magnitude, compared to gradient-based optimization methods. This paper is associated to code for reproducible research available at https://hal.inria.fr/hal-03552956