A Self-supervised Strategy for the Robustness of VQA Models - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

A Self-supervised Strategy for the Robustness of VQA Models

Jingyu Su
  • Fonction : Auteur
  • PersonId : 1275148
Chuanhao Li
Chenchen Jing
  • Fonction : Auteur
  • PersonId : 1275150
Yuwei Wu
  • Fonction : Auteur
  • PersonId : 1275151

Résumé

In visual question answering (VQA), most existing models suffer from language biases which make models not robust. Recently, many approaches have been proposed to alleviate language biases by generating samples for the VQA task. These methods require the model to distinguish original samples from synthetic samples, to ensure that the model fully understands two modalities of both visual and linguistic information rather than just predicts answers based on language biases. However, these models are still not sensitive enough to changes of key information in questions. To make full use of the key information in questions, we design a self-supervised strategy to make the nouns of questions be focused for enhancing the robustness of VQA models. Its auxiliary training process, predicting answers for synthetic samples generated by masking the last noun in questions, alleviates the negative influence of language biases. Experiments conducted on VQA-CP v2 and VQA v2 datasets show that our method achieves better results than other VQA models.
Fichier sous embargo
Fichier sous embargo
0 5 14
Année Mois Jours
Avant la publication
mercredi 1 janvier 2025
Fichier sous embargo
mercredi 1 janvier 2025
Connectez-vous pour demander l'accès au fichier

Dates et versions

hal-04178725 , version 1 (08-08-2023)

Licence

Identifiants

Citer

Jingyu Su, Chuanhao Li, Chenchen Jing, Yuwei Wu. A Self-supervised Strategy for the Robustness of VQA Models. 12th International Conference on Intelligent Information Processing (IIP), May 2022, Qingdao, China. pp.290-298, ⟨10.1007/978-3-031-03948-5_23⟩. ⟨hal-04178725⟩
13 Consultations
1 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More