多模态大语言模型如何解决图像任务：透视视觉定位、任务推理与答案解码
How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
AI总结：
本文提出线性探针框架，通过三类提示变化分析 LLaVA 与 Qwen2-VL 各层功能，揭示其视觉定位、语义推理和答案解码的稳定阶段结构。
摘要：
多模态大语言模型（MLLM）已在广泛的视觉-语言任务中展现出强劲性能，但其内部处理动态仍缺乏充分研究。本文提出一种探测框架，用于系统分析 MLLM 如何在各层处理视觉和文本输入。我们使用标准化锚点问题，训练线性分类器，从每一层提取的 token 嵌入中预测细粒度视觉类别（例如犬种）。为揭示不同层的功能角色，我们在三类受控提示变化下评估这些探针：（1）词汇变体，用于测试模型对表层变化的敏感性；（2）语义否定变体，通过修改提示中的视觉概念来翻转预期答案；（3）输出格式变体，在保持推理不变的同时改变答案格式。将该框架应用于 LLaVA-1.5、LLaVA-Next-LLaMA-3 和 Qwen2-VL，我们识别出一致的阶段性结构：早期层执行视觉定位，中间层支持词汇整合与语义推理，最后层准备任务特定输出。我们进一步表明，尽管整体阶段结构在视觉 token 化、指令微调数据和预训练语料变化下保持稳定，但各阶段对应的具体层分配会随基础 LLM 架构变化而显著移动。研究结果为 MLLM 的层级组织提供了统一视角，并提供了一种轻量级、模型无关的多模态表征动态分析方法。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated strong performance across a wide range of vision-language tasks, yet their internal processing dynamics remain underexplored. In this work, we introduce a probing framework to systematically analyze how MLLMs process visual and textual inputs across layers. We train linear classifiers to predict fine-grained visual categories (e.g., dog breeds) from token embeddings extracted at each layer, using a standardized anchor question. To uncover the functional roles of different layers, we evaluate these probes under three types of controlled prompt variations: (1) lexical variants that test sensitivity to surface-level changes, (2) semantic negation variants that flip the expected answer by modifying the visual concept in the prompt, and (3) output format variants that preserve reasoning but alter the answer format. Applying our framework to LLaVA-1.5, LLaVA-Next-LLaMA-3, and Qwen2-VL, we identify a consistent stage-wise structure in which early layers perform visual grounding, middle layers support lexical integration and semantic reasoning, and final layers prepare task-specific outputs. We further show that while the overall stage-wise structure remains stable across variations in visual tokenization, instruction tuning data, and pretraining corpus, the specific layer allocation to each stage shifts notably with changes in the base LLM architecture. Our findings provide a unified perspective on the layer-wise organization of MLLMs and offer a lightweight, model-agnostic approach for analyzing multimodal representation dynamics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
