LongVT: 通过原生工具调用激励'通过长视频思考'
LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling
AI总结：
本文提出LongVT，一种端到端的代理框架，通过交错的多模态工具链式思考实现'通过长视频思考'，通过利用LMM的固有时间定位能力作为原生视频裁剪工具，以解决长视频推理中的幻觉问题，并通过VideoSIAH数据集提升训练和评估效果。
摘要：
大型多模态模型（LMMs）在视频推理中展示出巨大的潜力，尤其是在文本链式思考（Chain-of-Thought）的应用中。然而，它们在处理长视频时仍然容易产生幻觉，尤其是当证据稀少且时间分布分散时。受人类理解长视频的方式启发——首先全局浏览，然后检查相关片段以获取细节——我们引入LongVT，一种端到端的代理框架，通过交错的多模态链式工具思考实现'通过长视频思考'。具体而言，我们利用LMM固有的时间定位能力作为原生视频裁剪工具，以聚焦特定视频片段并重新采样更细粒度的视频帧。这种从全局到局部的推理循环会持续进行，直到答案基于检索到的视觉证据得到支撑。鉴于长视频推理任务中细粒度问题-答案（QA）数据稀缺，我们整理并计划发布一个名为VideoSIAH的数据集，以促进训练和评估。具体而言，我们的训练数据集包含247.9万样本用于工具集成的冷启动监督微调，1.6千样本用于代理强化学习，以及15.4千样本用于代理强化学习微调。我们的评估基准包含1,280对精心挑选的QA对，通过半自动数据管道和人工在环验证进行筛选。通过精心设计的三阶段训练策略和广泛的实证验证，LongVT在四个具有挑战性的长视频理解和推理基准上均优于现有强大的基线。我们的代码、数据和模型检查点在https://github.com/EvolvingLMMs-Lab/LongVT上公开可用。
英文摘要：
Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
