A well-known issue in blind convolutive source separation is that the sources and filters are at best identifiable up to an arbitrary scaling and permutation at each frequency bin. We propose to exploit the sparsity of the filters as a consistency measure for correcting such permutations. We show that the permutation is well-posed, up to a global permutation, under appropriate sparsity hypotheses on the filters. A greedy combinatorial algorithm is proposed for permutation recovery. Its empirical performance shows that the time-domain sparsity of the filters allows to recover permutations much beyond theoretical predictions.