Knee OsteoArthritis (OA) is a common musculoskeletal disorder, which causes reduced mobility for seniors. Due to the semi-quantitative nature of the Kellgren-Lawrence (KL) grading system, medical practitioners’ grading is subjective, being entirely based on their experience. With the development of computer vision, Computer-Aided Diagnosis (CAD) systems based on deep learning methods such as convolutional neural networks (CNNs) have shown success in knee OA diagnosis. In this paper, we propose a new approach, the so-called Siamese-GAP Network, for the early detection of knee OA through a KL-grade classification. More precisely, a set of Global Average Pooling (GAP) layers is integrated into the Siamese network used to extract features from each level. The obtained features are then combined to improve the classification performance. Our experimental results on baseline X-ray images from the OsteoArthritis Initiative (OAI) dataset show that the proposed approach presents potential results for the early detection of knee OA.