arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.09425cs.CV
收藏

视觉语言模型是否需要处理图像标记?

Do Vision Language Models Need to Process Image Tokens?

IBM;Indian Institute of Science(印度科学研究所);University of Virginia(弗吉尼亚大学)

Sambit Ghosh, R. Venkatesh Babu, Chirag Agarwal

AI总结:

本文研究了视觉语言模型中图像标记的功能,发现视觉表示快速收敛至有限复杂度,而文本表示持续重构。视觉表示在不同层间可互换,任务依赖性影响视觉处理的必要性。

摘要:

视觉语言模型(VLMs)通过整合视觉编码器和大型语言模型(LLMs)取得了显著成功。尽管VLMs在深度变换器堆栈中处理密集图像标记(导致计算开销大),但其性能是否依赖持续的图像标记处理仍不清楚。本文系统研究了VLMs中图像标记的功能,发现视觉表示快速收敛至有限复杂度,即熵稳定、内在维度压缩、轨迹曲率接近常数。相比之下,文本表示持续重构。一旦稳定,视觉表示在不同层间可互换,表明深层阶段的额外变换有限。进一步,深度-wise视觉截断显示,视觉处理的必要性取决于任务:单标记预测对截断视觉深度较稳健,但多标记生成需持续访问视觉表示。在确定性解码中,减少视觉深度会更强烈地扰动中间推理轨迹而非最终输出,表明图像标记影响推理结构而非最终结论。这些发现质疑了VLMs中深层视觉处理统一必要的假设,挑战了当前多模态LLM架构范式。

英文摘要:

Vision Language Models (VLMs) have achieved remarkable success by integrating visual encoders with large language models (LLMs). While VLMs process dense image tokens across deep transformer stacks (incurring substantial computational overhead), it remains fundamentally unclear whether sustained image-token processing is necessary for their performance or visual representations meaningfully evolve from early to later layers. In this work, we systematically investigate the functional role of image tokens in VLMs and show that visual representations rapidly converge to a bounded-complexity regime, \ie their entropy stabilizes, intrinsic dimensionality compresses, and trajectory curvature approaches a near-constant profile. In contrast, textual representations continue to undergo substantial restructuring across depth. Once stabilized, visual representations become largely interchangeable between layers, indicating limited additional transformation in deeper stages. Further, depth-wise visual truncation reveals that the necessity of visual processing is task-dependent, where single-token predictions remain comparatively robust to truncated visual depth, but multi-token generation require sustained access to visual representations. Under deterministic decoding, reducing visual depth perturbs intermediate reasoning trajectories more strongly than final outputs, suggesting that image tokens influence the structure of reasoning more than the ultimate conclusions. Collectively, these findings \textbf{question the assumption} that deeper visual processing is uniformly essential in VLMs, challenging the current paradigm of multimodal LLM architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑