arXiv 2412.096162024-12-13cs.CV

V2PE：通过可变视觉位置编码提升视觉语言模型的多模态长上下文能力

V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding

University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）

Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, Xizhou Zhu

备注：: The code and models will be available at https://github.com/OpenGVLab/V2PE

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视觉语言模型长上下文能力不足的问题，提出可变视觉位置编码V2PE，结合增强长上下文数据集微调InternVL2，可处理最长1M token的多模态序列，在多类任务上表现优异。

摘要：

视觉语言模型（Vision-Language Models, VLMs）在处理各类多模态任务中展现出可观的能力，但在长上下文场景下表现不佳，尤其是在涉及视频、高分辨率图像或长篇图文文档的任务中。在本研究中，我们首先利用增强的长上下文多模态数据集，对VLM的长上下文能力开展了实证分析。研究结果表明，直接将文本token所用的位置编码机制应用于视觉token并非最优方案，且当位置编码超出模型的上下文窗口时，VLM的性能会急剧下降。为解决这一问题，我们提出了可变视觉位置编码（Variable Visual Position Encoding, V2PE），这是一种新型位置编码方法，它为视觉token采用可变且更小的增量，能够更高效地处理长多模态序列。我们的实验证明了V2PE的有效性，它能增强VLM有效理解和推理长多模态上下文的能力。我们进一步将V2PE与增强的长上下文多模态数据集结合，对开源VLM InternVL2进行微调。经过微调的模型在标准多模态任务和长上下文多模态任务上均取得了优异性能。值得注意的是，当训练数据集的序列长度提升至256K token时，该模型能够处理最长达1M token的多模态序列，凸显了其在实际长上下文应用中的潜力。

英文摘要：

Vision-Language Models (VLMs) have shown promising capabilities in handling various multimodal tasks, yet they struggle in long-context scenarios, particularly in tasks involving videos, high-resolution images, or lengthy image-text documents. In our work, we first conduct an empirical analysis of the long-context capabilities of VLMs using our augmented long-context multimodal datasets. Our findings reveal that directly applying the positional encoding mechanism used for textual tokens to visual tokens is suboptimal, and VLM performance degrades sharply when the position encoding exceeds the model's context window. To address this, we propose Variable Visual Position Encoding (V2PE), a novel positional encoding approach that employs variable and smaller increments for visual tokens, enabling more efficient management of long multimodal sequences. Our experiments demonstrate the effectiveness of V2PE to enhances VLMs' ability to effectively understand and reason over long multimodal contexts. We further integrate V2PE with our augmented long-context multimodal datasets to fine-tune the open-source VLM, InternVL2. The fine-tuned model achieves strong performance on both standard and long-context multimodal tasks. Notably, when the sequence length of the training dataset is increased to 256K tokens, the model is capable of processing multimodal sequences up to 1M tokens, highlighting its potential for real-world long-context applications.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。