以视觉为中心的Transformer重新审视视听分割
Revisiting Audio-Visual Segmentation with Vision-Centric Transformer
AI总结：
针对现有以音频为中心的Transformer在视听分割中存在的感知模糊和视觉细节丢失问题，提出以视觉为中心的VCT框架及PPQG模块，利用视觉派生查询聚合视听信息，在AVSBench数据集上取得最优性能。
摘要：
视听分割（AVS）旨在根据相关联的音频信号分割视频帧中发声的物体。现有的AVS方法通常采用以音频为中心的Transformer架构，其中物体查询由音频特征派生。然而，以音频为中心的Transformer存在两个局限性：由音频混合特性引起的感知模糊，以及因视觉细节丢失而削弱的密集预测能力。为了解决这些局限性，我们提出了一种新的以视觉为中心的Transformer（VCT）框架，该框架利用视觉派生的查询迭代地获取相应的音频和视觉信息，使查询能够更好地区分混合音频中的不同发声物体并准确描绘其轮廓。此外，我们还在VCT框架内引入了原型提示查询生成（PPQG）模块，通过音频原型提示和像素上下文分组，生成既具有语义感知又具有丰富视觉信息的视觉派生查询，从而促进视听信息聚合。大量实验表明，我们的VCT框架在AVSBench数据集的三个子集上取得了新的最先进性能。代码可在 https://github.com/spyflying/VCT_AVS 获取。
英文摘要：
Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where object queries are derived from audio features. However, audio-centric Transformers suffer from two limitations: perception ambiguity caused by the mixed nature of audio, and weakened dense prediction ability due to visual detail loss. To address these limitations, we propose a new Vision-Centric Transformer (VCT) framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information, enabling queries to better distinguish between different sounding objects from mixed audio and accurately delineate their contours. Additionally, we also introduce a Prototype Prompted Query Generation (PPQG) module within our VCT framework to generate vision-derived queries that are both semantically aware and visually rich through audio prototype prompting and pixel context grouping, facilitating audio-visual information aggregation. Extensive experiments demonstrate that our VCT framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset. The code is available at https://github.com/spyflying/VCT_AVS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
