Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance

Abstract

We propose our solution to the multimodal semantic role labeling task from the CON-STRAINT’22 workshop. The task aims at clas-sifying entities in memes into classes such as “hero” and “villain”. We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and implement three systems to classify the role of the entities. We propose dynamic sampling strategies to tackle the issue of class imbalance. Finally, we per-form qualitative analysis on the representations of the entities.

Dates and versions

hal-03840060 , version 1 (04-11-2022)

Identifiers

Cite

Syrielle Montariol, Étienne Simon, Arij Riabi, Djamé Seddah. Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance. CONSTRAINT 2022 - Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations, May 2022, Dublin, Ireland. pp.55-65, ⟨10.18653/v1/2022.constraint-1.7⟩. ⟨hal-03840060⟩
33 View
0 Download

Altmetric

Share

Gmail Facebook X LinkedIn More