Achieving high detection accuracy and high inference speed is important for a pedestrian detection system in self-driving applications. There exists a trade-off between detection accuracy and inference speed in modern convolu-tional object detectors. In this paper, we propose a novel pedestrian detection system, which leverages spatial attention and a two-level cascade of classification and bounding box regression to balance the trade-off. Our proposed spatial attention module reduces the search space for pedestrians by selecting a small set of anchor boxes for further processing. Furthermore, we present a two-level cascade of bounding box classification and regression and demonstrate its effectiveness for improved accuracy. We demonstrate the performance of our system on 2 public datasets-caltech-reasonable and citypersons; with state-of-art performance. Our ablation studies confirm the usefulness of our spatial attention and cascade modules.