TimeLens2：使用多模态大语言模型进行通用视频时间定位
TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs
浏览论文内容
中文总结 AI 辅助
研究通用视频时间定位问题，TimeLens2将时间证据视为区间集，通过多种策略构建多跨度监督，其时间瓦瑟斯坦奖励等方法提供反馈，在多个基准测试中表现出色，不同变体均有性能提升。
中文摘要 AI 辅助
视频多模态大语言模型能描述视频中发生的事情，但很少能确定支持证据出现的时间。我们研究通用视频时间定位，即一个模型预测跨视频长度、领域、查询形式和视角的可变基数证据区间集。现有训练策略与该集值任务不匹配。TimeLens2在整个监督和优化过程中将时间证据视为区间集。TimeLens2-93K通过字幕衍生提案、独立定位、跨智能体共识、语义验证和边界细化构建可靠的多跨度监督。我们的时间瓦瑟斯坦奖励计算合并区间支持上均匀分布之间精确的一维\(W_1\)，在不等基数和等效碎片化情况下提供密集、无匹配反馈；时间交并比用精确重叠反馈进行补充。在七个基准测试中，TimeLens2-2B在每个基准上均优于所有大小匹配的基线，4B和8B变体达到了当前最优性能。2B、4B和8B变体分别比其Qwen3-VL主干提高了14.2、13.0和18.1的平均交并比分数。
英文摘要
Video multimodal large language models (MLLMs) can describe what happens in a video, but rarely identify when the supporting evidence occurs. We study generalist video temporal grounding, in which one model predicts a variable-cardinality set of evidence intervals across video lengths, domains, query forms, and viewpoints. Existing training strategies are misaligned with this set-valued task: long-video labels often rely on brittle one-pass annotation, while reinforcement-learning rewards either fail to distinguish non-overlapping predictions or require fragile segment matching. TimeLens2 treats temporal evidence as an interval set throughout supervision and optimization. TimeLens2-93K constructs reliable multi-span supervision through caption-derived proposals, independent localization, cross-agent consensus, semantic verification, and boundary refinement. Our temporal Wasserstein reward computes exact one-dimensional \(W_1\) between uniform distributions over merged interval supports, providing dense, matching-free feedback under unequal cardinalities and equivalent fragmentation; temporal IoU complements it with precise-overlap feedback. Across seven benchmarks, TimeLens2-2B outperforms all size-matched baselines on every benchmark, while the 4B and 8B variants achieve state-of-the-art performance, surpassing open-source models with up to 397B parameters. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.
发表机构
- Nanjing University（南京大学）
- Shanghai AI Laboratory（上海人工智能实验室）
- Shanghai Jiao Tong University（上海交通大学）
- Zhejiang University（浙江大学）
- University of Science and Technology of China（中国科学技术大学）
- Fudan University（复旦大学）
机构由 AI 辅助整理，请以论文原文为准。
