VLM内部伪装视觉上下文的隐藏演化
The Hidden Evolution of Disguised Visual Context inside the VLM
AI总结：
研究视觉语言模型中视觉令牌如何通过不同集成架构（上下文注入与逐层注入）转化为有意义表示，揭示其内部演化过程及对性能的影响。
摘要：
视觉令牌作为原始的外部信号进入大语言模型（LLM）。它们如何被转化为有意义的表示并与语言空间交互完全取决于集成架构——无论是将视觉令牌视为输入序列中的上下文提示，还是直接注入到LLM的中间层。对于这些架构选择如何影响视觉信息及其内部转换以与LLM集成，目前仍缺乏受控比较和理解。我们通过在相同训练条件下评估上下文注入和逐层注入的VLM集成范式，在单图像、多图像和视频基准上进行公平比较。在此过程中，我们揭示了一个隐藏的演化：视觉令牌作为伪装的视觉上下文（缺乏语言结构的原始表示）进入LLM，但根据集成范式逐渐被重塑，每种范式捕捉视觉信号的不同频率特征。我们表明，LLM内部的这种演化决定了VLM能够有效利用哪些视觉特征、视觉表示如何与语言空间对齐，以及最终每种范式在不同任务上的表现。我们进一步证明，仅关注注意力分配是不够的，性能由每一层视觉表示的质量驱动。
英文摘要：
Visual tokens enter Large Language Models (LLMs) as raw, foreign signals. How they are transformed into meaningful representations and interact with the language space depends entirely on the integration architecture. Whether by treating visual tokens as in-context prompts within the input sequence or injecting them directly into the LLM's intermediate layers. A controlled comparison and understanding of how these architectural choices affect visual information and its internal transformation to integrate with the LLM remains underexplored. We provide a fair comparison by evaluating in-context and layer-wise injection VLM integration paradigms under identical training conditions across single image, multi-image, and video benchmarks. In doing so, we uncover a hidden evolution where visual tokens enter the LLM as disguised visual context, raw representations lacking linguistic structure, but are progressively reshaped depending on the integration paradigm, each capturing fundamentally different frequency characteristics of the visual signal. We show that this evolution inside the LLM determines what visual features the VLM can utilize effectively, how visual representations align with the language space, and ultimately how each paradigm performs across different tasks. We further demonstrate that attention allocation alone is insufficient, and that performance is driven by the quality of visual representations at each layer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
