MovieChat+：面向长视频问答的问题感知稀疏记忆
MovieChat+: Question-aware Sparse Memory for Long Video Question Answering
AI总结：
本文提出MovieChat，基于Atkinson-Shiffrin记忆模型与稀疏记忆机制，以零样本方式提升多模态大模型对长视频的理解能力，并发布MovieChat-1K基准验证其有效性。
摘要：
最近，将视频基础模型与大型语言模型相结合以构建视频理解系统，能够克服特定预定义视觉任务的局限性。然而，现有方法要么采用复杂的时空模块，要么严重依赖额外的感知模型来提取用于视频理解的时序特征，并且它们仅在短视频上表现良好。对于长视频，与长期时序连接相关的计算复杂度和内存成本显著增加，带来了额外的挑战。利用 Atkinson-Shiffrin 记忆模型，以 Transformers 中的 token 作为记忆载体，并结合我们专门设计的记忆机制，我们提出 MovieChat 来克服这些挑战。我们利用零样本方法，提升预训练多模态大型语言模型以理解长视频，而不引入额外的可训练时序模块。MovieChat 在长视频理解方面实现了最先进的性能，同时发布了 MovieChat-1K 基准，包含 1K 长视频、2K 时序定位标签和 14K 人工标注，以验证我们方法的有效性。代码和数据集可通过以下网址访问：https://github.com/rese1f/MovieChat。
英文摘要：
Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, and they only perform well on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose MovieChat to overcome these challenges. We lift pre-trained multi-modal large language models for understanding long videos without incorporating additional trainable temporal modules, employing a zero-shot approach. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video, 2K temporal grounding labels, and 14K manual annotations for validation of the effectiveness of our method. The code along with the dataset can be accessed via the following https://github.com/rese1f/MovieChat.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
