VideoITG: 多模态视频理解中的指导性时间定位
VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding
AI总结：
VideoITG通过设计VidThinker流水线，自动生成指令条件化描述，检索相关视频片段并选择关键帧，提升多模态视频理解任务的性能。
摘要：
尽管视频大语言模型（Video-LLMs）在多模态理解和推理任务中展现出显著潜力，但如何高效地从视频中选择最有信息量的帧仍是一个关键挑战。现有方法试图通过减少帧间冗余或采用无监督事件定位来优化帧采样，但这些方法在处理复杂指令遵循任务和需要精确时间建模的场景时往往效果有限，导致语义对齐和时间推理能力受限。为了解决上述挑战，我们引入了针对视频的指导性时间定位（VideoITG）框架，旨在根据用户指令自适应地定制帧采样策略。具体而言，我们设计了VidThinker流水线，通过自动生成指令条件化的描述、检索相关视频片段并选择关键帧，实现高效的监督。使用VidThinker，我们构建了包含40,000个视频和500,000个时间定位注释的VideoITG-40K数据集。我们的插件式VideoITG模型利用Video-LLMs的视觉-语言对齐和推理能力进行判别性帧选择。VideoITG在多个多模态视频理解基准测试中 consistently 提升了性能，展示了其有效性和潜力。
英文摘要：
While Video Large Language Models (Video-LLMs) have shown significant potential in multimodal understanding and reasoning tasks, how to efficiently select the most informative frames from videos remains a critical challenge. Existing methods attempt to optimize frame sampling by reducing inter-frame redundancy or employing unsupervised event localization. However, these approaches often fall short in handling complex instruction-following tasks and scenarios that demand precise temporal modeling, resulting in limited performance in both semantic alignment and temporal reasoning. To address the above challenges, we introduce Instructed Temporal Grounding for Videos (VideoITG), a framework aiming to adaptively customize frame sampling strategies based on user instructions. Specifically, we design the VidThinker pipeline, which automates annotation by generating instruction-conditioned captions, retrieving relevant video segments, and selecting key frames to enable efficient supervision. Using VidThinker, we build the VideoITG-40K dataset with 40K videos and 500K temporal grounding annotations. Our plug-and-play VideoITG model leverages Video-LLMs' visual-language alignment and reasoning for discriminative frame selection. VideoITG consistently boosts the performance on multiple multimodal video understanding benchmarks, demonstrating its effectiveness and potential.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
