通过关键语义感知线索增强音视频问答
Boosting Audio Visual Question Answering via Key Semantic-Aware Cues
AI总结：
本文提出时空感知模型TSPM，通过构建陈述性句子提示和空间感知模块，有效感知音视频问答中关键的时空视听线索，在多个AVQA基准上表现优异。
摘要：
音视频问答（AVQA）任务旨在回答与视频中各种视觉对象、声音及其交互相关的问题。这类自然的多模态视频包含丰富且复杂的动态音视频组件，其中只有一部分与给定问题密切相关。因此，有效感知与给定问题相关的音视频线索对于正确回答至关重要。在本文中，我们提出了一种时空感知模型（TSPM），旨在使模型能够感知与问题相关的关键视觉和听觉线索。具体而言，考虑到使用视觉-语言预训练模型将非陈述性问题和视觉表示对齐到同一语义空间的挑战，我们构建了源自问题模板的陈述性句子提示，以协助时间感知模块更好地识别与问题相关的关键片段。随后，设计了一个空间感知模块，用于合并来自所选片段的视觉标记以突出关键潜在目标，随后与音频进行跨模态交互以感知潜在的声觉感知区域。最后，整合来自这些模块的重要时空线索来回答问题。在多个AVQA基准上的大量实验表明，我们的框架不仅在理解音视频场景方面表现出色，而且在有效回答复杂问题方面也表现优异。代码可在https://github.com/GeWu-Lab/TSPM获取。
英文摘要：
The Audio Visual Question Answering (AVQA) task aims to answer questions related to various visual objects, sounds, and their interactions in videos. Such naturally multimodal videos contain rich and complex dynamic audio-visual components, with only a portion of them closely related to the given questions. Hence, effectively perceiving audio-visual cues relevant to the given questions is crucial for correctly answering them. In this paper, we propose a Temporal-Spatial Perception Model (TSPM), which aims to empower the model to perceive key visual and auditory cues related to the questions. Specifically, considering the challenge of aligning non-declarative questions and visual representations into the same semantic space using visual-language pretrained models, we construct declarative sentence prompts derived from the question template, to assist the temporal perception module in better identifying critical segments relevant to the questions. Subsequently, a spatial perception module is designed to merge visual tokens from selected segments to highlight key latent targets, followed by cross-modal interaction with audio to perceive potential sound-aware areas. Finally, the significant temporal-spatial cues from these modules are integrated to answer the question. Extensive experiments on multiple AVQA benchmarks demonstrate that our framework excels not only in understanding audio-visual scenes but also in answering complex questions effectively. Code is available at https://github.com/GeWu-Lab/TSPM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
