视觉标记究竟编码了什么？揭示多模态大语言模型中的稀疏性与冗余性
What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
AI总结：
本研究揭示多模态大语言模型中视觉标记的稀疏性与冗余性，通过EmbedLens工具发现活跃标记在进入模型前已编码细粒度信息，并提出中层注入方法提升效率。
摘要：
多模态大语言模型（MLLMs）将视觉标记投影到语言模型的嵌入空间中，但其内部结构和视觉语义的处理仍缺乏深入理解。在本工作中，我们引入了一个双重视角分析框架，包含一种新的探测工具EmbedLens，用于进行细致分析。我们发现输入层面存在显著的语义稀疏性：视觉标记一致分为汇入、死亡和活跃三类。值得注意的是，只有活跃标记，约占总输入的60%，才包含图像特定的意义。此外，通过一个针对补丁压缩的基准测试，我们证明这些活跃标记在进入LLM之前已经编码了丰富的、细粒度的线索（例如物体、颜色和OCR）。内部视觉计算（如视觉注意力和前馈网络）对于大多数标准任务来说是冗余的。对于一小部分高度视觉导向的任务，实际上受益于内部处理，我们揭示了活跃标记自然与中间LLM层对齐，而不是初始嵌入空间，表明浅层处理是不必要的，直接中层注入既足够。最终，我们的发现提供了一个统一的视觉标记处理机制观点，通过选择性标记剪枝、最小化视觉计算和中层注入，为更高效和可解释的MLLM架构铺平了道路。代码已发布在：https://github.com/EIT-NLP/EmbedLens。
英文摘要：
Multimodal large language models (MLLMs) project visual tokens into the embedding space of language models, yet the internal structuring and processing of visual semantics remain poorly understood. In this work, we introduce a two-fold analytical framework featuring a novel probing tool, $\textbf{EmbedLens}$, to conduct a fine-grained analysis. We uncover a pronounced semantic sparsity at the input level: visual tokens consistently partition into sink, dead, and alive categories. Remarkably, only the alive tokens, comprising $\approx60\%$ of the total input, carry image-specific meaning. Furthermore, using a targeted patch-compression benchmark, we demonstrate that these alive tokens already encode rich, fine-grained cues (e.g., objects, colors, and OCR) prior to entering the LLM. Internal visual computations (such as visual attention and feed-forward networks) are redundant for most standard tasks. For the small subset of highly vision-centric tasks that actually benefit from internal processing, we reveal that alive tokens naturally align with intermediate LLM layers rather than the initial embedding space, indicating that shallow-layer processing is unnecessary and that direct mid-layer injection is both sufficient. Ultimately, our findings provide a unified mechanistic view of visual token processing, paving the way for more efficient and interpretable MLLM architectures through selective token pruning, minimized visual computation, and mid-layer injection. The code is released at: https://github.com/EIT-NLP/EmbedLens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
