Tarsier2：从详细视频描述到全面视频理解，推进大型视觉-语言模型
Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding
AI总结：
Tarsier2通过扩展预训练数据、时间对齐和DPO优化，在视频描述与理解任务上超越GPT-4o等模型，并在15个基准上达到最优。
摘要：
我们推出了Tarsier2，这是一种最先进的大型视觉-语言模型（LVLM），专为生成详细且准确的视频描述而设计，同时展现出卓越的通用视频理解能力。Tarsier2通过三项关键升级取得了显著进展：（1）将预训练数据从1100万扩展到4000万视频-文本对，丰富了数据量和多样性；（2）在监督微调期间执行细粒度的时间对齐；（3）使用基于模型的采样自动构建偏好数据，并应用DPO训练进行优化。大量实验表明，Tarsier2-7B在详细视频描述任务中持续优于领先的专有模型，包括GPT-4o和Gemini 1.5 Pro。在DREAM-1K基准上，Tarsier2-7B的F1分数比GPT-4o提高了2.8%，比Gemini-1.5-Pro提高了5.8%。在人工并排评估中，Tarsier2-7B相比GPT-4o显示出+8.6%的性能优势，相比Gemini-1.5-Pro显示出+24.9%的优势。Tarsier2-7B还在15个公共基准上创造了新的最先进结果，涵盖视频问答、视频定位、幻觉测试和具身问答等任务，展示了其作为稳健通用视觉-语言模型的多功能性。
英文摘要：
We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM) designed for generating detailed and accurate video descriptions, while also exhibiting superior general video understanding capabilities. Tarsier2 achieves significant advancements through three key upgrades: (1) Scaling pre-training data from 11M to 40M video-text pairs, enriching both volume and diversity; (2) Performing fine-grained temporal alignment during supervised fine-tuning; (3) Using model-based sampling to automatically construct preference data and applying DPO training for optimization. Extensive experiments show that Tarsier2-7B consistently outperforms leading proprietary models, including GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K benchmark, Tarsier2-7B improves F1 by 2.8% over GPT-4o and 5.8% over Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6% performance advantage over GPT-4o and +24.9% over Gemini-1.5-Pro. Tarsier2-7B also sets new state-of-the-art results across 15 public benchmarks, spanning tasks such as video question-answering, video grounding, hallucination test, and embodied question-answering, demonstrating its versatility as a robust generalist vision-language model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
