多模态大语言模型中的视觉功能层
Vision Function Layer in Multimodal LLMs
AI总结：
研究提出Visual Token Swapping框架识别MLLM中的视觉功能层VFL，并据此设计VFL-LoRA与VFL-select，实现防遗忘微调及高效数据选择。
摘要：
本研究发现，在多模态大语言模型（MLLM）中，与视觉相关的功能解码分布在不同的解码器层。通常，每项功能（例如计数、定位或OCR识别）都会收敛到两到三层，我们将其定义为视觉功能层（VFL）。此外，不同VFL的深度及其顺序在不同MLLM之间呈现出一致模式，并且与人类行为高度吻合（例如，识别最先发生，随后是计数，然后是定位）。这些发现来自Visual Token Swapping（视觉令牌交换），这是我们提出的新型分析框架，通过修改目标KV cache条目来精确阐明解码过程中特定层的功能。此外，这些洞见在为真实世界下游应用定制MLLM方面具有显著实用性。例如，当LoRA训练被选择性地应用于功能与训练数据相匹配的VFL时，VFL-LoRA不仅优于full-LoRA，还能避免域外功能遗忘。此外，通过分析特定VFL被消融时训练数据上的性能差异，VFL-select会按功能自动分类数据，从而实现高效的数据选择，直接增强相应能力。因此，VFL-select在数据选择上超越人类专家，并且仅使用原始数据集20%的数据即可达到全量数据性能的98%。本研究增进了对MLLM视觉处理的理解，推动构建更高效、可解释且鲁棒的模型。
英文摘要：
This study identifies that visual-related functional decoding is distributed across different decoder layers in Multimodal Large Language Models (MLLMs). Typically, each function, such as counting, grounding, or OCR recognition, narrows down to two or three layers, which we define as Vision Function Layers (VFL). Additionally, the depth and its order of different VFLs exhibits a consistent pattern across different MLLMs, which is well-aligned with human behaviors (e.g., recognition occurs first, followed by counting, and then grounding). These findings are derived from Visual Token Swapping, our novel analytical framework that modifies targeted KV cache entries to precisely elucidate layer-specific functions during decoding. Furthermore, these insights offer substantial utility in tailoring MLLMs for real-world downstream applications. For instance, when LoRA training is selectively applied to VFLs whose functions align with the training data, VFL-LoRA not only outperform full-LoRA but also prevent out-of-domain function forgetting. Moreover, by analyzing the performance differential on training data when particular VFLs are ablated, VFL-select automatically classifies data by function, enabling highly efficient data selection to directly bolster corresponding capabilities. Consequently, VFL-select surpasses human experts in data selection, and achieves 98% of full-data performance with only 20% of the original dataset. This study delivers deeper comprehension of MLLM visual processing, fostering the creation of more efficient, interpretable, and robust models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
