Severe cognitive impairment involving multiple cognitive domains can occur early during the course of multiple sclerosis (MS). We investigated resting state functional connectivity changes in large-scale brain networks and related structural damage underlying cognitive dysfunction in patients with early MS. Patients with relapsing MS (3-5 years disease duration) were prospectively assigned to two groups based on a standardized neuropsychological evaluation: (1) cognitively impaired group (CI group, n = 15), with abnormal performances in at least 3 tests; (2) cognitively preserved group (CP group, n = 20) with normal performances in all tests. Patients and age-matched healthy controls underwent a multimodal 3T magnetic resonance imaging (MRI) including anatomical T1 and T2 images, diffusion imaging and resting state functional MRI. Structural MRI analysis revealed that CI patients had a higher white matter lesion load compared to CP and a more severe atrophy in gray matter regions highly connected to networks involved in cognition. Functional connectivity measured by integration was increased in CP patients versus controls in attentional networks (ATT), while integration was decreased in CI patients compared to CP both in the default mode network (DMN) and ATT. An anatomofunctional study within the DMN revealed that functional connectivity was mostly altered between the medial prefrontal cortex (MPFC) and the posterior cingulate cortex (PCC) in CI patients compared to CP and controls. In a multilinear regression model, functional correlation between MPFC and PCC was best predicted by PCC atrophy. Disconnection in the DMN and ATT networks may deprive the brain of compensatory mechanisms required to face widespread structural damage.