检索以恢复：通过语义一致的净化实现不完整音频-视觉问答
Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification
AI总结：
本文提出R$^{2}$ScP框架，通过检索而非生成方法处理缺失模态，利用统一语义嵌入进行跨模态检索，并引入上下文感知适应净化机制，提升语义恢复能力，实验表明其在不完整模态场景中性能更优。
摘要：
近年来，音频-视觉问答（AVQA）方法取得了显著进展。然而，大多数AVQA方法缺乏有效处理缺失模态的机制，在数据中断的现实场景中表现不佳。此外，处理缺失模态的现有方法大多依赖生成填补来合成缺失特征。虽然部分有效，但这些方法倾向于捕捉跨模态共同性，却难以获取缺失数据中的独特、模态特定知识，导致幻觉和推理准确性下降。为解决这些挑战，我们提出R$^{2}$ScP，一种新的框架，将处理缺失模态的范式从传统生成填补转向基于检索的恢复。具体而言，我们利用统一语义嵌入进行跨模态检索以获取缺失领域知识。为了最大化语义恢复，我们引入一种上下文感知的适应性净化机制，以消除检索数据中的潜在语义噪声。此外，我们采用两阶段训练策略，以显式建模不同来源知识之间的语义关系。广泛实验表明，R$^{2}$ScP显著提高了AVQA性能，并增强了在模态不完整场景中的鲁棒性。
英文摘要：
Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R$^{2}$ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R$^{2}$ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
