Age, sex and APOE-ε4 genotype have been identified as the strongest predictors of the risk of developing Alzheimer's Disease (AD). It is well documented that women's cognitive decline and neurodegeneration happen faster compared to men in AD, while being comparatively better preserved in healthy populations. This work analyses regional cortical thinning and brain atrophy over time using non-linear mixed-effect models, and assesses the association of both sex and APOE-ε4 genotype with the onset and pace of atrophy in each region. Statistical comparisons allow to isolate the effect of both those genetic covariates and to provide a brain map of the regions whose atrophic patterns are most affected. This study is provided for both AD progression and healthy ageing. We found that female sex is a stronger risk factor for neurodegeneration than APOE-ε4 genotype in AD, and women experience much faster atrophic rates, but slightly later onset, after correcting for the APOE status. APOE-ε4 genotype lead to earlier and faster atrophy for most regions of the brain for AD patients but not for healthy patients. Healthy patients with higher education experience slightly delayed atrophy but not patients with neurodegeneration. Lastly, for a cohort of patients with mild cognitive impairment, sex shows the same association as for the cognitively normal cohort, while APOE-ε4 genotype shows the same association as in the AD cohort.