Alzheimer’s Disease (AD) is a heterogeneous disease that disproportionately affects women and people with the APOE-ε4 susceptibility gene. We aim to describe the not-well-understood influence of both risk factors on the dynamics of brain atrophy in AD and healthy aging. Regional cortical thinning and brain atrophy were modeled over time using non-linear mixed-effect models and the FreeSurfer software. Covariance analysis was used to disentangle the effect of sex and APOE genotype on the regional onset age and pace of atrophy, while correcting for educational level. A map of the regions most affected by neurodegeneration is provided. Results were confirmed on gray matter density data from the SPM software. Women experience faster atrophic rates in the temporal, frontal, parietal lobes and limbic system and earlier onset in the amygdalas, but slightly later onset in the postcentral and cingulate gyri as well as all regions of the basal ganglia and thalamus. APOE-ε4 genotypes leads to earlier and faster atrophy in the temporal, frontal, parietal lobes and limbic system in AD patients, but not in healthy patients. Higher education was found to slightly delay atrophy in healthy patients, but not for AD patients. A cohort of amyloid positive patients with MCI showed a similar impact of sex as in the healthy cohort, while APOE-ε4 showed similar associations as in the AD cohort. Female sex is as strong a risk factor for AD as APOE−ε4 genotype regarding neurodegeneration. Women experience a sharper atrophy in the later stages of the disease although not a significantly earlier onset. These findings may have important implications for the development of targeted intervention.