缓解视觉引导音频字幕中的视听不匹配问题
Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning
AI总结：
针对真实场景中视觉引导音频字幕的视听不匹配问题，提出熵感知门控融合框架与批量视听洗牌技术，有效抑制误导性视觉线索并增强模型鲁棒性，在AudioCaps基准上性能提升且推理速度约快6倍。
摘要：
当前视觉引导音频字幕系统常无法处理真实场景中的视听不对齐，如配音内容或画外音。为填补此关键空白，我们提出熵感知门控融合框架，通过跨模态不确定性量化动态调节视觉信息流。该新方法在交叉注意力层采用注意力熵分析，自动识别并抑制模态融合中的误导性视觉线索。作为架构补充，我们开发了批量视听洗牌技术，生成合成不匹配训练对，极大增强模型对对齐噪声的鲁棒性。在AudioCaps基准上的评估表明，系统性能优于现有基线，尤其在模态不匹配场景中。此外，相比基线，推理速度提升约6倍。
英文摘要：
Current vision-guided audio captioning systems frequently fail to address audiovisual misalignment in real-world scenarios, such as dubbed content or off-screen sounds. To bridge this critical gap, we present an entropy-aware gated fusion framework that dynamically modulates visual information flow through cross-modal uncertainty quantification. Our novel approach employs attention entropy analysis in cross-attention layers to automatically identify and suppress misleading visual cues during modal fusion. Complementing this architecture, we develop a batch-wise audiovisual shuffling technique that generates synthetic mismatched training pairs, greatly enhancing model resilience against alignment noise. Evaluations on the AudioCaps benchmark demonstrate our system's superior performance over existing baselines, especially in mismatched modality scenarios. Furthermore, our solution demonstrates an approximately 6x improvement in inference speed compared to the baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
