arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.03384cs.CV
收藏

LongVLM:通过大型语言模型实现高效的长视频理解

LongVLM: Efficient Long Video Understanding via Large Language Models

Monash University(莫纳什大学);MBZUAI;University of Science and Technology of China(中国科学技术大学)

Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, Bohan Zhuang

AI总结:

LongVLM通过将长视频分解为短期片段并融合局部与全局特征,实现高效长视频理解,优于现有方法。

摘要:

借助大型语言模型(LLMs),近期基于视频的大型语言模型(VideoLLMs)的进展推动了各种视频理解任务的发展。这些模型通过对大量视觉令牌进行池化或查询聚合来编码视频表示,从而使得计算和内存成本变得可承受。尽管成功地提供了对视频内容的整体理解,但现有的VideoLLMs在处理长期视频时,由于忽略了局部信息,仍然面临实现细节理解的挑战。为了解决这一挑战,我们引入了LongVLM,一个简单而强大的用于长视频理解的VideoLLM,其基于一个观察:长视频通常由连续的关键事件、复杂动作和摄像机运动组成。我们的方法提出将长视频分解为多个短期片段,并通过一个分层令牌合并模块为每个片段编码局部特征。这些特征按时间顺序拼接,以保持跨连续短期片段的故事线。此外,我们提出将全局语义整合到每个局部特征中,以增强上下文理解。通过这种方式,我们编码了包含局部和全局信息的视频表示,使LLM能够为长期视频生成全面的响应。在VideoChatGPT基准和零样本视频问答数据集上的实验结果表明,我们的模型优于之前的最先进方法。定性示例显示,我们的模型为长视频理解产生了更精确的响应。代码可在https://github.com/ziplab/LongVLM获取。

英文摘要:

Empowered by Large Language Models (LLMs), recent advancements in Video-based LLMs (VideoLLMs) have driven progress in various video understanding tasks. These models encode video representations through pooling or query aggregation over a vast number of visual tokens, making computational and memory costs affordable. Despite successfully providing an overall comprehension of video content, existing VideoLLMs still face challenges in achieving detailed understanding due to overlooking local information in long-term videos. To tackle this challenge, we introduce LongVLM, a simple yet powerful VideoLLM for long video understanding, building upon the observation that long videos often consist of sequential key events, complex actions, and camera movements. Our approach proposes to decompose long videos into multiple short-term segments and encode local features for each segment via a hierarchical token merging module. These features are concatenated in temporal order to maintain the storyline across sequential short-term segments. Additionally, we propose to integrate global semantics into each local feature to enhance context understanding. In this way, we encode video representations that incorporate both local and global information, enabling the LLM to generate comprehensive responses for long-term videos. Experimental results on the VideoChatGPT benchmark and zero-shot video question-answering datasets demonstrate the superior capabilities of our model over the previous state-of-the-art methods. Qualitative examples show that our model produces more precise responses for long video understanding. Code is available at https://github.com/ziplab/LongVLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑