arXiv 2604.051172026-04-06cs.CVcs.AIcs.CL

在回答前观看：从视觉引导的后训练中学习

Watch Before You Answer: Learning from Visually Grounded Post-Training

University of British Columbia（不列颠哥伦比亚大学）；Vector Institute（向量研究所）；Etude AI；Kolors Team, Kuaishou Technology（快手科技Kolors团队）；University of Toronto（多伦多大学）；University of Waterloo（滑铁卢大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang, Penghui Du, Yiming Jia, Dongfu Jiang, Xuan He, Shenhui Zhang, Ping Nie, Peter West, Kelsey R. Allen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文发现现有视频理解基准中40-60%的问题可通过文本线索回答，提出VidGround方法通过仅使用视觉引导问题提升VLM性能，实验显示其在后训练中效果优于复杂方法。

摘要：

对于视觉-语言模型（VLMs）来说，全面理解视觉、时间与文本线索至关重要。然而，尽管多模态建模取得进展，视频理解性能仍落后于基于文本的推理。本文发现进展甚至比预期更差：广泛报告的长视频理解基准中40-60%的问题可通过文本线索单独回答。此外，这些问题也普遍存在在广泛使用的后训练数据集中，可能削弱后训练提升VLM视频理解性能的能力。受此观察启发，我们引入VidGround作为简单有效的解决方案：仅使用实际视觉引导问题进行后训练。当与基于强化学习的后训练算法结合使用时，该简单技术可将性能提升6.2个点，仅使用原始数据集的69.1%。此外，我们展示数据整理与简单后训练算法优于复杂方法，表明数据质量是提升VLM视频理解能力的主要瓶颈。这些结果强调了整理后训练数据和评估基准的重要性，以推动更强大VLM的发展。项目页面：http://vidground.etuagi.com。

英文摘要：

It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: http://vidground.etuagi.com.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。