Grounded-VideoLLM：增强视频大语言模型中的细粒度时序定位能力
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models
AI总结：
本文提出 Grounded-VideoLLM，通过额外时序流和离散时序 token 增强时间建模与时间戳表示，并采用多阶段训练及自动标注的 grounded VideoQA 数据集，显著提升视频大语言模型的细粒度时序定位能力。
摘要：
视频大语言模型（Video-LLMs）在粗粒度视频理解方面已展现出卓越能力，但在细粒度时序定位上仍面临困难。本文提出 Grounded-VideoLLM，一种能够以细粒度方式感知和推理特定视频片段的新型 Video-LLM。我们发现当前 Video-LLMs 在细粒度视频理解上存在局限，原因是缺乏有效的时序建模和时间戳表示。为此，我们通过引入（1）额外的时序流来编码帧间关系，以及（2）富含特定时间知识的离散时序 token 来表示时间戳，从而增强模型能力。为优化 Grounded-VideoLLM 的训练，我们采用多阶段训练方案，从简单的视频描述任务开始，逐步引入复杂度递增的视频时序定位任务。为进一步提升 Grounded-VideoLLM 的时序推理能力，我们还通过自动标注流程构建了一个带定位信息的 VideoQA 数据集。大量实验表明，Grounded-VideoLLM 不仅在时序句子定位、密集视频描述和带定位信息的 VideoQA 等细粒度定位任务中表现优异，还展现出作为通用视频理解多功能视频助手的巨大潜力。
英文摘要：
Video Large Language Models (Video-LLMs) have demonstrated remarkable capabilities in coarse-grained video understanding, however, they struggle with fine-grained temporal grounding. In this paper, we introduce Grounded-VideoLLM, a novel Video-LLM adept at perceiving and reasoning over specific video moments in a fine-grained manner. We identify that current Video-LLMs have limitations for fine-grained video understanding since they lack effective temporal modeling and timestamp representation. In light of this, we sharpen our model by incorporating (1) an additional temporal stream to encode the relationships between frames and (2) discrete temporal tokens enriched with specific time knowledge to represent timestamps. To optimize the training of Grounded-VideoLLM, we employ a multi-stage training scheme, beginning with simple video-captioning tasks and progressively introducing video temporal grounding tasks of increasing complexity. To further enhance Grounded-VideoLLM's temporal reasoning capability, we also curate a grounded VideoQA dataset by an automatic annotation pipeline. Extensive experiments demonstrate that Grounded-VideoLLM not only excels in fine-grained grounding tasks such as temporal sentence grounding, dense video captioning, and grounded VideoQA, but also shows great potential as a versatile video assistant for general video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
