QDFormer：基于量化语义分解的复杂环境下鲁棒音视频分割
QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition
AI总结：
提出基于乘积量化的语义分解方法，将多源音频分解为单源语义，并引入全局到局部量化机制，显著提升复杂环境下音视频分割性能。
摘要：
音视频分割（AVS）是一项具有挑战性的任务，旨在根据视频中物体关联的声学线索分割视觉对象。由于涉及多个声源和背景干扰，在音频与视觉内容之间建立稳健的对应关系面临独特挑战，原因包括：（1）声源之间的复杂纠缠，（2）不同声音事件发生频率的频繁变化。假设声音事件独立发生，多源语义空间可表示为单源子空间的笛卡尔积。我们受到启发，将多源音频语义分解为单源语义，以便与视觉内容进行更有效的交互。我们提出了一种基于乘积量化的语义分解方法，其中多源语义可被分解并表示为若干解耦且抑制噪声的单源语义。此外，我们引入了一种从全局到局部的量化机制，将稳定的全局（片段级）特征中的知识蒸馏到局部（帧级）特征中，以处理音频语义的频繁变化。大量实验表明，我们语义分解后的音频表示显著提升了AVS性能，例如，在具有ResNet50骨干的具有挑战性的AVS-Semantic基准上，mIoU提升了+21.2%。https://github.com/lxa9867/QSD。
英文摘要：
Audiovisual segmentation (AVS) is a challenging task that aims to segment visual objects in videos according to their associated acoustic cues. With multiple sound sources and background disturbances involved, establishing robust correspondences between audio and visual contents poses unique challenges due to (1) complex entanglement across sound sources and (2) frequent changes in the occurrence of distinct sound events. Assuming sound events occur independently, the multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We are motivated to decompose the multi-source audio semantics into single-source semantics for more effective interactions with visual content. We propose a semantic decomposition method based on product quantization, where the multi-source semantics can be decomposed and represented by several disentangled and noise-suppressed single-source semantics. Furthermore, we introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones, to handle frequent changes in audio semantics. Extensive experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance, e.g., +21.2% mIoU on the challenging AVS-Semantic benchmark with ResNet50 backbone. https://github.com/lxa9867/QSD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
