窄门：原生多模态模型中的局部图像-文本通信
The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
AI总结：
研究揭示了原生多模态模型中图像与文本信息交互的机制，发现通过单个标记作为窄门影响图像理解性能，提出基于标记级干预的精细控制方法。
摘要：
最近多模态训练的进展显著提高了统一模型中图像理解和生成的整合程度。本研究探讨了视觉语言模型（VLMs）如何处理图像理解任务，重点在于如何处理和转移视觉信息到文本领域。我们比较了原生多模态VLMs，即从头开始在多模态数据上训练以生成文本和图像的模型，与非原生多模态VLMs，即从预训练大语言模型中适应或只能生成文本的模型，突出信息流动的关键差异。我们发现，在原生多模态VLMs中，图像和文本嵌入在残差流中更加分离。此外，VLMs在视觉信息如何到达文本方面存在差异：非原生多模态VLMs表现出分布式的通信模式，其中信息通过多个图像标记进行交换，而专门针对联合图像和文本生成训练的模型则倾向于依赖一个后图像标记，作为视觉信息的窄门。我们显示，删除这个单一标记会显著降低图像理解性能，而针对的标记级干预可以可靠地引导图像语义和下游文本，通过精细控制实现。
英文摘要：
Recent advances in multimodal training have significantly improved the integration of image understanding and generation within a unified model. This study investigates how vision-language models (VLMs) handle image-understanding tasks, focusing on how visual information is processed and transferred to the textual domain. We compare native multimodal VLMs, models trained from scratch on multimodal data to generate both text and images, and non-native multimodal VLMs, models adapted from pre-trained large language models or capable of generating only text, highlighting key differences in information flow. We find that in native multimodal VLMs, image and text embeddings are more separated within the residual stream. Moreover, VLMs differ in how visual information reaches text: non-native multimodal VLMs exhibit a distributed communication pattern, where information is exchanged through multiple image tokens, whereas models trained natively for joint image and text generation tend to rely on a single post-image token that acts as a narrow gate for visual information. We show that ablating this single token significantly deteriorates image-understanding performance, whereas targeted, token-level interventions reliably steer image semantics and downstream text with fine-grained control.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
