In this paper, we present a deep learning based approach to performing the whole-day prediction of the traffic speed for the elevated highway. In order to learn the temporal features of traffic speed data in a hierarchical way, an improved convolutional neural network (CNN) with asymmetric kernels is proposed. Speed data are collected from loop detectors of Yan’an elevated highway of Shanghai. To test the performance of the presented method, we compare it with some conventional approaches of traffic speed estimation. Experimental results demonstrate that our method outperforms all of them.