Many spectral-spatial classification methods of HSI based on convolutional neural network (CNN) are proposed and achieve outstanding performance recently. However, these methods require tremendous computations with complex network and excessively large model. Moreover, single machine is obviously weak when dealing with big data. In this paper, a parallel dimensionality-varied convolutional neural network (DV-CNN) is proposed to address these issues. The dimensionalities of feature maps extracted vary with stages in DV-CNN, and DV-CNN reduces the dimensionalities of feature maps to simplify the computation and the structure of network without information loss. Besides, the parallel architecture of DV-CNN can obviously reduce the training time. The experiments compared with state-of-the-art methods are performed on Indian Pines and Pavia University scene datasets. The results of experiments demonstrate that parallel DV-CNN can obtain better classification performance, reduce the time consuming and improve the training efficiency.