基于查询引导的时空频交互的音乐音频视觉问答
Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering
AI总结：
本文提出一种查询引导的时空频交互方法，通过整合问题引导的线索和频域特性，提升音频-视觉问答的性能。
摘要：
音频-视觉问答（AVQA）是一个具有挑战性的多模态任务，需要在给定的视频中联合推理音频、视觉和文本信息以回答自然语言问题。受最近视频问答（Video QA）进展的启发，许多现有的AVQA方法主要集中在视觉信息处理上，利用预训练模型提取物体级和运动级表示。然而，在这些方法中，音频输入主要被当作视频分析的补充，而文本问题信息对音频-视觉理解的贡献很小，因为通常只在推理的最后阶段整合。为了解决这些限制，我们提出了一种新颖的查询引导的时空频（QSTar）交互方法，该方法有效整合了问题引导的线索，并利用音频信号的频域特性，结合空间和时间感知，以增强音频-视觉理解。此外，我们引入了一个受提示启发的查询上下文推理（QCR）块，该块引导模型更精确地关注语义相关的音频和视觉特征。在多个AVQA基准上的大量实验表明，我们提出的方法的有效性，取得了显著优于现有音频问答、视觉问答、视频问答和AVQA方法的性能提升。代码和预训练模型将在发表后发布。
英文摘要：
Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on two AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code is released under https://github.com/lik1996/QSTar.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
