arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.18512cs.CVcs.AI

FCoT-VL:通过高效视觉令牌压缩推进面向文本的大型视觉语言模型

FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression

University of Science and Technology of China(中国科学技术大学)

Jianjian Li, Junquan Fan, Feng Tang, Gang Huang, Shitao Zhu, Songlin Liu, Nian Xie, Wulong Liu, Yong Liao

AI总结:

本文提出FCoT-VL框架,通过轻量自蒸馏预训练和高质量后训练压缩视觉令牌,在保持性能的同时降低计算开销,并在面向文本基准上超越基线。

摘要:

视觉大语言模型(VLLMs)的快速成功往往依赖于包含大量视觉令牌的高分辨率图像,这阻碍了训练和部署的效率。当前免训练的视觉令牌压缩方法在高分辨率、面向文本的图像理解与推理任务中表现出严重的性能下降。在本文中,我们提出了一种面向文本的VLLMs在高分辨率场景下的高效视觉令牌压缩框架。具体而言,我们采用一个轻量级的自蒸馏预训练阶段来压缩视觉令牌,仅需有限的图像-文本对和最少的可学习参数。随后,为了缓解令牌压缩模型可能出现的性能下降,我们构建了一个高质量的后期训练阶段。为验证我们方法的有效性,我们将其应用于先进的VLLMs——InternVL2。实验结果表明,我们的方法显著降低了计算开销,同时在一系列面向文本的基准测试中优于基线方法。我们将很快发布模型和代码。

英文摘要:

The rapid success of Vision Large Language Models (VLLMs) often depends on the high-resolution images with abundant visual tokens, which hinders training and deployment efficiency. Current training-free visual token compression methods exhibit serious performance degradation in tasks involving high-resolution, text-oriented image understanding and reasoning. In this paper, we propose an efficient visual token compression framework for text-oriented VLLMs in high-resolution scenarios. In particular, we employ a light-weight self-distillation pre-training stage to compress the visual tokens, requiring a limited numbers of image-text pairs and minimal learnable parameters. Afterwards, to mitigate potential performance degradation of token-compressed models, we construct a high-quality post-train stage. To validate the effectiveness of our method, we apply it to an advanced VLLMs, InternVL2. Experimental results show that our approach significantly reduces computational overhead while outperforming the baselines across a range of text-oriented benchmarks. We will release the models and code soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑