Due to structural and functional brain variability between subjects, cortical atlases may contain errors, such as a label mismatch across subjects. Indeed, a particular region of one subject might not correspond structurally or functionally to the region with the same label of another subject. Previous work showed that it is possible to increase the similarity between brain network connectivity matrices computed from diffusion Magnetic Resonance Imaging (MRI) using a graph matching approach. However, graph matching has never been considered on brain networks derived from function MRI. In this work, we present the results of graph matching using both structural and functional networks. Fast Approximate Quadratic Assignment Problem (FAQ) algorithm has been used to tackle the NP-hard problem of graph matching, in minimizing the Frobenius distance between the connectivity matrices to the Frechet mean connectivity matrix of the group. We modified the FAQ algorithm in order to find a single permutation matrix applied on both structural and functional matrices minimizing the two distances to their respective Frechet mean connectome, including a weight term α to favor one modality over the other. The matrices are normalized because each modality has its own range of values. We observe that matching networks using only one modality (α = 0 or 1) does not improve the similarity of the other modality. However, there is a trade-off for α = 0.72, where we improved the structural or functional similarity in more than 80% of the subjects. In summary, our results show that we can find a permutation matrix that increases the similarity for both structural and functional brain networks.