Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain

Résumé

Functional magnetic resonance imaging (fMRI) can be used to map patterns of brain activity and understand how information is expressed in the human brain. Using fMRI data to analyses the relationship between visual cortex and language semantic representation is of significance for building a new deep learning model. The cognition of visual semantic concept refers to the behavior that people can distinguish and classify the semantic concept of visual information they see. Many previous research literatures have revealed semantically active brain regions, but lack of modeling the relationship between visual information and language semantic concept in human brain, which is very important to understand the brain mechanism of concept learning. In this paper, we propose a Semantic Concept Cognitive Network (S-ConceptNet) model of brain cortical signals based on fMRI, The model organizes visual and linguistic semantic information into a unified representation framework, which can effectively analyses the generation process of semantic concept, and realize the function of semantic concept cognition. Based on S-ConceptNet, we also use the Dual-learning model to reconstruct the brain signal, judge the corresponding concept category through the S-ConceptNet, compare the reconstructed image with the image of this category. And finally output semantic information corresponding to the brain signal through similarity. We verify the effect of the model and conduct comparative experiments, and the experimental results are better than previous work, and prove the effectiveness of the model proposed in this paper.
Fichier sous embargo
Fichier sous embargo
0 5 16
Année Mois Jours
Avant la publication
mercredi 1 janvier 2025
Fichier sous embargo
mercredi 1 janvier 2025
Connectez-vous pour demander l'accès au fichier

Dates et versions

hal-04317198 , version 1 (01-12-2023)

Licence

Identifiants

Citer

Haodong Jing, Ming Du, Yongqiang Ma, Nanning Zheng. Exploring the Relationship Between Visual Information and Language Semantic Concept in the Human Brain. 18th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2022, Hersonissos, Greece. pp.394-406, ⟨10.1007/978-3-031-08333-4_32⟩. ⟨hal-04317198⟩
19 Consultations
1 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More