Convolutional matrix factorization (ConvMF), which integrates convolutional neural network (CNN) into probabilistic matrix factorization (PMF), has been recently proposed to utilize the contextual information and achieve higher rating prediction accuracy of model-based collaborative filtering (CF) recommender systems. While ConvMF uses max-pooling, which may lose the feature’s location and frequency information. In order to solve this problem, a novel approach with segmented max-pooling (ConvMF-S) has been proposed in this paper. ConvMF-S can extract multiple features and keep their location and frequency information. Experiments show that the rating prediction accuracy has been improved.