arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.09907cs.CV
收藏

IQViC:面向长期视频理解LMMs的上下文、问题自适应视觉压缩器

IQViC: In-context, Question Adaptive Vision Compressor for Long-term Video Understanding LMMs

Fujitsu Research, Fujitsu Limited(富士通研究所,富士通株式会社)

Sosuke Yamao, Natsuki Miyahara, Yuki Harazono, Shun Takeuchi

AI总结:

针对长期视频理解中现有方法性能下降和内存开销大的问题,提出基于Transformer的上下文、问题自适应视觉压缩器IQViC,通过选择性提取相关信息减少内存令牌,在InfiniBench等基准上提升理解准确率和内存效率。

摘要:

随着视频数据复杂性的增加以及对更高效长期时间理解的需求,现有的长期视频理解方法往往难以准确捕获和分析长时间的视频序列。这些方法通常难以在更长的持续时间上保持性能,也难以处理视频内容中错综复杂的依赖关系。为解决这些限制,我们提出了一种简单而有效的用于长期视频理解的大型多模态模型框架,该框架引入了一种新颖的视觉压缩器,即上下文、问题自适应视觉压缩器(IQViC)。其关键思想,受人类选择性注意和上下文记忆机制的启发,是引入一种新颖的视觉压缩器并结合高效的内存管理技术,以增强长期视频问答。我们的框架利用基于Transformer的视觉压缩器IQViC,实现问题条件下的上下文压缩,这与依赖完整视频视觉特征的现有方法不同。这能选择性地提取相关信息,显著减少内存令牌需求。通过在基于InfiniBench的新数据集以及用于现有方法评估的标准基准上进行的大量实验,我们证明了所提出的IQViC框架的有效性,以及其在视频理解准确性和内存效率方面优于最先进方法的性能。

英文摘要:

With the increasing complexity of video data and the need for more efficient long-term temporal understanding, existing long-term video understanding methods often fail to accurately capture and analyze extended video sequences. These methods typically struggle to maintain performance over longer durations and to handle the intricate dependencies within the video content. To address these limitations, we propose a simple yet effective large multi-modal model framework for long-term video understanding that incorporates a novel visual compressor, the In-context, Question Adaptive Visual Compressor (IQViC). The key idea, inspired by humans' selective attention and in-context memory mechanisms, is to introduce a novel visual compressor and incorporate efficient memory management techniques to enhance long-term video question answering. Our framework utilizes IQViC, a transformer-based visual compressor, enabling question-conditioned in-context compression, unlike existing methods that rely on full video visual features. This selectively extracts relevant information, significantly reducing memory token requirements. Through extensive experiments on a new dataset based on InfiniBench for long-term video understanding, and standard benchmarks used for existing methods' evaluation, we demonstrate the effectiveness of our proposed IQViC framework and its superiority over state-of-the-art methods in terms of video understanding accuracy and memory efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑