阅读，而非思考：理解并弥合多模态大语言模型中文本变为像素时的模态差距
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
AI总结：
本文系统诊断多模态大语言模型在处理图像文本时的模态差距，发现其源于模型推理意愿不足而非感知失败，并提出一种轻量级自蒸馏方法有效弥合该差距。
摘要：
多模态大语言模型（MLLMs）能够处理以图像形式呈现的文本，但它们的表现往往不如相同内容以文本令牌形式提供时。我们通过在五种输入模式下跨七个基准评估七个MLLM，系统性地诊断了这种“模态差距”，涵盖了从合成渲染文本到来自arXiv PDF和Wikipedia页面的真实文档图像。我们发现，该差距对字体和分辨率等渲染选择高度敏感，并且自然文档图像通常表现出更小的差距，这表明性能差异部分反映了评估伪影而非根本性限制。通过对超过4000个示例进行基于扎根理论的错误分析，我们确定了主要原因：仅图像输入抑制了推理努力，模型产生的输出短5-19倍，跳过了逐步计算或推理。不愿推理，而非感知或知识检索失败，驱动了性能差距，尤其是在需要多步推理的任务上。我们展示了一种简单的、轻量级的在线自蒸馏方法，通过让模型在其自身的文本模式推理轨迹与图像输入配对上进行微调，弥合了这一差距，将图像模式准确率提升至匹配或超过文本模式性能，提升超过50%，并且增益可迁移到未见过的基准而不会灾难性遗忘。总体而言，我们的结果和分析提供了对模态差距的系统理解，并指出了在多模态语言模型中改进视觉文本理解的实际路径。
英文摘要：
Multimodal large language models (MLLMs) can process text presented as images, yet they often perform worse than when the same content is provided as textual tokens. We systematically diagnose this "modality gap" by evaluating seven MLLMs across seven benchmarks in five input modes, spanning both synthetically rendered text and realistic document images from arXiv PDFs to Wikipedia pages. We find that the gap is highly sensitive to rendering choices such as font and resolution, and that natural document images often exhibit much smaller gaps, suggesting the performance difference partly reflects evaluation artifacts rather than fundamental limitations. Through a grounded-theory error analysis of over 4,000 examples, we identify the primary cause: image input alone suppresses reasoning effort, with models producing 5--19x shorter outputs that skip step-by-step computation or reasoning. The reluctance to reason, not a failure of perception or knowledge retrieval, drives the performance gap, particularly on tasks requiring multi-step reasoning. We show that a simple, lightweight on-policy self-distillation method by fine-tuning models on their own text-mode reasoning traces paired with image inputs closes this gap, raising image-mode accuracy to match or exceed text-mode performance with over 50\% improvement, and the gains transfer to unseen benchmarks without catastrophic forgetting. Overall, our results and analyses provide a systematic understanding of the modality gap and suggest a practical path toward improving visual text understanding in multimodal language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
