Lane boundary detection is a key technology for self-driving cars. In this paper, we propose a spatiotemporal, deep learning based lane boundary detection method that can accurately detect lane boundaries under complex weather conditions and traffic scenarios in real time. Our algorithm consists of three parts: (i) inverse perspective transform and lane boundary position estimation using the spatial and temporal constraints of lane boundaries, (ii) convolutional neural networks (CNN) based boundary type classification and position regression, (iii) optimization and lane fitting. Our algorithm is designed to accurately detect lane boundaries and classify line types under a variety of environment conditions in real time. We tested our proposed algorithm on three open- source datasets and also compared the results with other state-of-the-art methods. Experimental results showed that our algorithm achieved high accuracy and robustness for detecting lane boundaries in a variety of scenarios in real time. Besides, we also realized the application of our algorithm on embedded platforms and verified the algorithm’s real-time performance on real self-driving cars.