Robust Audio-Visual Instance Discrimination via Active Contrastive Set Mining - Inria - Institut national de recherche en sciences et technologies du numérique Access content directly
Conference Papers Year : 2022

Robust Audio-Visual Instance Discrimination via Active Contrastive Set Mining

Abstract

The recent success of audio-visual representation learning can be largely attributed to their pervasive property of audio-visual synchronization, which can be used as self-annotated supervision. As a state-of-the-art solution, Audio-Visual Instance Discrimination (AVID) extends instance discrimination to the audio-visual realm. Existing AVID methods construct the contrastive set by random sampling based on the assumption that the audio and visual clips from all other videos are not semantically related. We argue that this assumption is rough, since the resulting contrastive sets have a large number of faulty negatives. In this paper, we overcome this limitation by proposing a novel Active Contrastive Set Mining (ACSM) that aims to mine the contrastive sets with informative and diverse negatives for robust AVID. Moreover, we also integrate a semantically-aware hard-sample mining strategy into our ACSM. The proposed ACSM is implemented into two most recent state-of-the-art AVID methods and significantly improves their performance. Extensive experiments conducted on both action and sound recognition on multiple datasets show the remarkably improved performance of our method.

Dates and versions

hal-03906947 , version 1 (19-12-2022)

Licence

Attribution

Identifiers

Cite

Hanyu Xuan, Yihong Xu, Shuo Chen, Zhiliang Wu, Jian Yang, et al.. Robust Audio-Visual Instance Discrimination via Active Contrastive Set Mining. IJCAI 2022 - 31st International Joint Conference on Artificial Intelligence, Jul 2022, Vienna, Austria. pp.3643-3649, ⟨10.24963/ijcai.2022/506⟩. ⟨hal-03906947⟩
29 View
0 Download

Altmetric

Share

Gmail Facebook X LinkedIn More