SimpleOCR: 将可视化问题呈现以教导大语言模型阅读
SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read
AI总结:
SimpleOCR通过强制模型视觉参与,解决多模态大语言模型在图像中阅读文本的性能问题,提升模型的视觉文本提取能力。
摘要:
尽管多模态大语言模型(MLLMs)在快速发展,但关于其视觉定位机制仍存在一个关键问题未被回答:这些模型真正“阅读”嵌入在图像中的文本,还是仅仅依赖于文本提示中的参数捷径?在本文中,我们通过引入可视化问题(VQ)设置来诊断这一问题,其中文本查询直接呈现在图像上,以结构上强制视觉参与。我们在Qwen2.5-VL上进行的诊断实验揭示了一个惊人的能力利用差距:尽管具有强大的OCR能力,模型在VQ设置下的性能会降级高达12.7%,暴露了深层次的“模态懒惰”。为弥合这一差距,我们提出了SimpleOCR,一种即插即用的训练策略,它在学习过程中施加了结构约束。通过将训练样本转换为VQ格式并随机化风格,SimpleOCR有效否定了基于文本的捷径,迫使模型激活并优化其视觉文本提取路径。实验表明,SimpleOCR在不修改架构的情况下取得了稳健的提升。在四个代表性OOD基准测试中,它比基模型高出5.4%,比基于原始图像的GRPO高出2.7%,并且表现出极高的数据效率,仅使用30倍更少的样本(8.5K)就实现了优于最近RL方法的性能。此外,其即插即用的性质允许无缝集成到高级RL策略如NoisyRollout中,以获得互补的改进。代码可在https://github.com/aiming-lab/SimpleOCR上获得。
英文摘要:
Despite the rapid advancements in Multimodal Large Language Models (MLLMs), a critical question regarding their visual grounding mechanism remains unanswered: do these models genuinely ``read'' text embedded in images, or do they merely rely on parametric shortcuts in the text prompt? In this work, we diagnose this issue by introducing the Visualized-Question (VQ) setting, where text queries are rendered directly onto images to structurally mandate visual engagement. Our diagnostic experiments on Qwen2.5-VL reveal a startling capability-utilization gap: despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting, exposing a deep-seated ``modality laziness.'' To bridge this gap, we propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process. By transforming training samples into the VQ format with randomized styles, SimpleOCR effectively invalidates text-based shortcuts, compelling the model to activate and optimize its visual text extraction pathways. Empirically, SimpleOCR yields robust gains without architectural modifications. On four representative OOD benchmarks, it surpasses the base model by 5.4% and GRPO based on original images by 2.7%, while exhibiting extreme data efficiency, achieving superior performance with 30x fewer samples (8.5K) than recent RL-based methods. Furthermore, its plug-and-play nature allows seamless integration with advanced RL strategies like NoisyRollout to yield complementary improvements. Code is available at https://github.com/aiming-lab/SimpleOCR.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。