This paper presents a novel low complexity time domain algorithm for blind separation of speech signal from their convolutive mixtures. We try to reduce intrinsic computational complexity of time domain algorithms by adapting only a small subset of taps from separating FIR filters which are expected to attain largest values. This selection is accomplished by recovering spatial dependencies using Linear Prediction (LP) analysis. Then we use Particle Swarm Optimization (PSO) in order to find best values for these selected taps. We employ the sparseness properties of speech signals in the Time-Frequency (TF) domain to define a low complexity and yet appropriate fitness function which numerically quantifies the amount of achieved separation by each one of the particles during PSO execution.