arXiv 2402.160502024-10-03cs.CVcs.CL

利用时间定位桥实现多模态大语言模型的高效时间外推

Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridge

Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；State Key Laboratory of General Artificial Intelligence（通用人工智能国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机技术研究院）

Yuxuan Wang, Yueqian Wang, Pengfei Wu, Jianxin Liang, Dongyan Zhao, Yang Liu, Zilong Zheng

备注：: To appear at EMNLP 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出时间定位桥（TGB）框架，通过多跨度时间定位算法、多模态长度外推训练和免标注引导桥接，显著提升多模态大语言模型的长视频时间理解能力，并在七个视频基准上验证了其可扩展性和有效性。

摘要：

尽管多模态大语言模型（MLLMs）取得了进展，但在响应语言查询时解读长视频的挑战依然存在，这主要归因于时间定位的低效以及预训练上下文窗口大小的限制。在本工作中，我们提出了时间定位桥（Temporal Grounding Bridge，TGB），这是一个新颖的框架，通过引导（bootstrapping）为MLLMs赋予先进的时间定位能力并扩展其上下文范围。我们的框架通过三项关键创新显著增强了当前MLLMs的时间能力：一种高效的多跨度时间定位算法，应用于从光流投影得到的低维时间特征；一种多模态长度外推训练范式，利用低维时间特征来扩展训练上下文窗口大小；以及一种引导框架，无需标注即可将我们的模型与可插拔的MLLMs桥接起来。我们在七个视频基准上验证了TGB，并展示了与先前MLLMs相比显著的性能提升。值得注意的是，我们的模型最初在四帧序列上训练，能够有效处理长达16倍的序列而不牺牲性能，突显了其在实际应用中的可扩展性和有效性。我们的代码已在 https://github.com/bigai-nlco/VideoTGB 公开。

英文摘要：

Despite progress in multimodal large language models (MLLMs), the challenge of interpreting long-form videos in response to linguistic queries persists, largely due to the inefficiency in temporal grounding and limited pre-trained context window size. In this work, we introduce Temporal Grounding Bridge (TGB), a novel framework that bootstraps MLLMs with advanced temporal grounding capabilities and broadens their contextual scope. Our framework significantly enhances the temporal capabilities of current MLLMs through three key innovations: an efficient multi-span temporal grounding algorithm applied to low-dimension temporal features projected from flow; a multimodal length extrapolation training paradigm that utilizes low-dimension temporal features to extend the training context window size; and a bootstrapping framework that bridges our model with pluggable MLLMs without requiring annotation. We validate TGB across seven video benchmarks and demonstrate substantial performance improvements compared with prior MLLMs. Notably, our model, initially trained on sequences of four frames, effectively handles sequences up to 16 longer without sacrificing performance, highlighting its scalability and effectiveness in real-world applications. Our code is publicly available at https://github.com/bigai-nlco/VideoTGB

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。