Brain-inspired computing is a research hotspot in artificial intelligence (AI). One of the key problems in this field is how to find the bridge between brain connectivity and data correlation in a connection-to-cognition model. Functional magnetic resonance imaging (fMRI) signals provide rich information about brain activities. Existing modeling approaches with fMRI focus on the strength information, but neglect structural information. In a previous work, we proposed a monolayer correlation network (CorrNet) to model the structural connectivity. In this paper, we extend the monolayer CorrNet to a hierarchical correlation network (HcorrNet) by analysing visual stimuli of natural images and fMRI signals in the entire visual cortex, that is, V1, V2 V3, V4, fusiform face area (FFA), the lateral occipital complex (LOC) and parahippocampal place area (PPA). Through the HcorrNet, the efficient connectivity of the brain can be inferred layer by layer. Then, the stimulus-sensitive activity mode of voxels can be extracted, and the forward encoding process of visual perception can be modeled. Both of them can guide the decoding process of fMRI signals, including classification and image reconstruction. In the experiments, we improved a dynamic evolving spike neuron network (SNN) as the classifier, and used Generative Adversarial Networks (GANs) to reconstruct image.