arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.19702cs.CVcs.AIcs.MM
收藏

TimeSuite:通过锚定调优提升多模态大语言模型的长视频理解能力

TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning

Nanjing University(南京大学);Shanghai AI Laboratory(上海人工智能实验室);SIAT, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Zhejiang University(浙江大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学)

Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, Yali Wang, Yu Qiao, Limin Wang

AI总结:

针对多模态大语言模型(MLLMs)长视频理解能力不足的问题,提出TimeSuite设计集合,通过长视频处理框架、锚定调优数据集及指令任务适配MLLMs,所构建的VideoChat-T模型在基准测试中性能提升显著,零样本锚定能力优异。

摘要:

多模态大语言模型(MLLMs)在短视频理解任务中展现出了令人印象深刻的性能。然而,对于MLLMs而言,理解长视频仍然具有挑战性。本文提出TimeSuite,这是一套新的设计集合,旨在将现有的短视频MLLMs适配到长视频理解任务中,包括一个简单高效的长视频序列处理框架、一个用于MLLMs锚定调优的高质量视频数据集,以及一个精心设计的指令调优任务,该任务将锚定监督以传统问答格式显式融入其中。具体而言,基于VideoChat,我们提出了长视频MLLM模型VideoChat-T,通过实现token混洗来压缩长视频token,并引入时间自适应位置编码(TAPE)以增强视觉表示的时间感知能力。同时,我们引入TimePro数据集,这是一个全面的以锚定为中心的指令调优数据集,包含9项任务和34.9万条高质量锚定标注。值得注意的是,我们设计了一种新的指令调优任务类型——时间锚定描述,用于生成包含对应时间戳预测的详细视频描述。这种显式的时间位置预测将引导MLLM在生成描述时正确关注视觉内容,从而降低由LLMs导致的幻觉风险。实验结果表明,TimeSuite为增强短视频MLLM的长视频理解能力提供了一种成功的解决方案,在Egoschema和VideoMME基准测试中分别实现了5.6%和6.8%的性能提升。此外,VideoChat-T展现出了强大的零样本时间锚定能力,显著优于现有的最先进MLLMs;经过微调后,其性能与传统的有监督专家模型相当。

英文摘要:

Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in short video understanding. However, understanding long-form videos still remains challenging for MLLMs. This paper proposes TimeSuite, a collection of new designs to adapt the existing short-form video MLLMs for long video understanding, including a simple yet efficient framework to process long video sequence, a high-quality video dataset for grounded tuning of MLLMs, and a carefully-designed instruction tuning task to explicitly incorporate the grounding supervision in the traditional QA format. Specifically, based on VideoChat, we propose our long-video MLLM, coined as VideoChat-T, by implementing a token shuffling to compress long video tokens and introducing Temporal Adaptive Position Encoding (TAPE) to enhance the temporal awareness of visual representation. Meanwhile, we introduce the TimePro, a comprehensive grounding-centric instruction tuning dataset composed of 9 tasks and 349k high-quality grounded annotations. Notably, we design a new instruction tuning task type, called Temporal Grounded Caption, to peform detailed video descriptions with the corresponding time stamps prediction. This explicit temporal location prediction will guide MLLM to correctly attend on the visual content when generating description, and thus reduce the hallucination risk caused by the LLMs. Experimental results demonstrate that our TimeSuite provides a successful solution to enhance the long video understanding capability of short-form MLLM, achieving improvement of 5.6% and 6.8% on the benchmarks of Egoschema and VideoMME, respectively. In addition, VideoChat-T exhibits robust zero-shot temporal grounding capabilities, significantly outperforming the existing state-of-the-art MLLMs. After fine-tuning, it performs on par with the traditional supervised expert models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑