我能信任你的答案吗？视觉定位的视频问答
Can I Trust Your Answer? Visually Grounded Video Question Answering
AI总结：
本文构建了带时间定位标签的 NExT-GQA 数据集，揭示现有视觉-语言模型在视频问答中缺乏视觉证据支撑的问题，并提出基于高斯掩码优化和跨模态学习的定位问答方法以提升模型可信度。
摘要：
针对利用预训练技术进行视频-语言理解的新兴趋势，我们研究了视觉定位的视频问答（VideoQA）。具体而言，通过迫使视觉-语言模型（VLMs）在回答问题的同时提供视觉证据，我们试图确定此类技术的预测在多大程度上真正锚定于相关视频内容，而非来自语言或不相关视觉上下文的虚假相关性。为此，我们构建了 NExT-GQA——它是 NExT-QA 的扩展，包含 10.5K 个与原始问答对相关联的时间定位（或位置）标签。借助 NExT-GQA，我们审视了一系列最先进的 VLMs。通过事后注意力分析，我们发现这些模型尽管在问答性能上表现强劲，但在为答案提供证据方面极其薄弱。这暴露了当前 VLMs 在做出可靠预测方面的局限性。作为补救措施，我们进一步探索并提出了一种通过高斯掩码优化和跨模态学习实现的定位问答方法。使用不同骨干网络的实验表明，这种定位机制同时改善了定位和问答性能。通过这些努力，我们旨在推动 VQA 系统中可信赖的 VLMs 的发展。我们的数据集和代码可在 https://github.com/doc-doc/NExT-GQA 获取。
英文摘要：
We study visually grounded VideoQA in response to the emerging trends of utilizing pretraining techniques for video-language understanding. Specifically, by forcing vision-language models (VLMs) to answer questions and simultaneously provide visual evidence, we seek to ascertain the extent to which the predictions of such techniques are genuinely anchored in relevant video content, versus spurious correlations from language or irrelevant visual context. Towards this, we construct NExT-GQA -- an extension of NExT-QA with 10.5$K$ temporal grounding (or location) labels tied to the original QA pairs. With NExT-GQA, we scrutinize a series of state-of-the-art VLMs. Through post-hoc attention analysis, we find that these models are extremely weak in substantiating the answers despite their strong QA performance. This exposes the limitation of current VLMs in making reliable predictions. As a remedy, we further explore and propose a grounded-QA method via Gaussian mask optimization and cross-modal learning. Experiments with different backbones demonstrate that this grounding mechanism improves both grounding and QA. With these efforts, we aim to push towards trustworthy VLMs in VQA systems. Our dataset and code are available at https://github.com/doc-doc/NExT-GQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
