arXiv 2407.166382024-07-23cs.CV

揭示与缓解音视频分割中的偏差

Unveiling and Mitigating Bias in Audio Visual Segmentation

Beijing University of Posts and Telecommunications（北京邮电大学）；Renmin University of China（中国人民大学）

Peiwen Sun, Honggang Zhang, Di Hu

备注：: Accepted by ACM MM 24 (ORAL)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究揭示了音视频分割中的音频启动偏差与视觉先验问题，通过引入音频感知模块和对比训练策略分别缓解，在AVS基准上取得有竞争力的性能。

摘要：

社区研究人员开发了一系列先进的音视频分割模型，旨在提高发声物体掩码的质量。尽管这些模型生成的掩码初看合理，但偶尔会表现出带有错误定位逻辑的异常。我们将此归因于现实世界固有的偏好和分布作为学习信号比复杂的音视频定位更简单，从而导致对重要模态信息的忽视。通常，异常现象往往复杂且无法直接系统地观察。在本研究中，我们使用适当的合成数据进行了开创性尝试，根据异常来源将现象分类并分析为“音频启动偏差”和“视觉先验”两种类型。针对音频启动偏差，为增强音频对不同强度和语义的敏感性，一个专门针对音频的感知模块感知潜在语义信息，并将信息纳入有限的查询集中，即主动查询。此外，transformer解码器中与此类主动查询相关的交互机制被定制，以适应音频语义间交互调节的需求。针对视觉先验，探索了多种对比训练策略，通过引入偏差分支来优化模型，甚至无需改变模型结构。在实验中，观察证明了现有模型偏差的存在及其产生的影响。最后，通过对AVS基准的实验评估，我们证明了我们的方法在处理两类偏差方面的有效性，在所有三个子集上取得了有竞争力的性能。

英文摘要：

Community researchers have developed a range of advanced audio-visual segmentation models aimed at improving the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding, which leads to the disregard of important modality information. Generally, the anomalous phenomena are often complex and cannot be directly observed systematically. In this study, we made a pioneering effort with the proper synthetic data to categorize and analyze phenomena as two types "audio priming bias" and "visual prior" according to the source of anomalies. For audio priming bias, to enhance audio sensitivity to different intensities and semantics, a perception module specifically for audio perceives the latent semantic information and incorporates information into a limited set of queries, namely active queries. Moreover, the interaction mechanism related to such active queries in the transformer decoder is customized to adapt to the need for interaction regulating among audio semantics. For visual prior, multiple contrastive training strategies are explored to optimize the model by incorporating a biased branch, without even changing the structure of the model. During experiments, observation demonstrates the presence and the impact that has been produced by the biases of the existing model. Finally, through experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of our methods in handling both types of biases, achieving competitive performance across all three subsets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文