多语言OCR感知微调和提示引导的链式思维推理用于多模态大语言模型
Multilingual OCR-Aware Fine-Tuning and Prompt-Guided Chain-of-Thought Reasoning for Multimodal Large Language Models
AI总结：
提出一种多语言OCR感知的多模态训练框架，通过合成数据生成、OCR感知微调和结构化视觉链式思维提示，提升多模态大语言模型在复杂视觉条件下的OCR完整性和多语言翻译准确性。
摘要：
光学字符识别（OCR）和多语言文本理解仍然是多模态大语言模型（MLLMs）的主要失败模式，尤其是在包含杂乱布局、小字体、模糊、遮挡和复杂排版的真实世界图像中。我们提出了一种OCR感知的多语言多模态训练框架，该框架结合了（i）大规模合成OCR到翻译数据生成，（ii）使用LoRA适配的OCR感知监督微调（SFT），以及（iii）在不确定视觉条件下进行推理的结构化视觉链式思维（CoT）提示。使用基于LLaMA的多模态架构，所提出的框架在OCR完整性、多语言翻译准确性和退化视觉条件下的鲁棒性方面有了显著提升。在多语言收据、菜单、海报、标志、手写文本和文档图像上的实验结果表明，与基线模型相比，视觉-文本对齐显著改善。特别是，所提出的OCR感知后训练框架提高了对小、模糊、空间分散和部分遮挡文本的提取，同时减少了对不确定OCR条件下语言先验的依赖。与前沿多模态系统（包括GPT-5类和Gemini系列模型）的定性比较进一步表明，在噪声和视觉模糊的OCR场景下，OCR对齐得到改善，幻觉减少。总体而言，结果表明，以数据为中心的OCR感知多模态后训练为改进多语言OCR和基于OCR的视觉问答系统提供了一种有效且可扩展的方向。
英文摘要：
Optical character recognition (OCR) and multilingual scene-text understanding remain challenging for multimodal large language models (MLLMs), particularly in real-world images containing small or degraded text, cluttered layouts, occlusion, handwriting, and complex typography. We present an OCR-aware multilingual post-training framework that improves visual-text grounding in a general-purpose MLLM without requiring an external OCR engine, OCR-extracted text, or text bounding boxes at inference time. The framework combines large-scale multilingual OCR supervision, approximately 5M additional multilingual training samples, controlled synthetic OCR generation and in-image text translation, LoRA-based supervised fine-tuning (SFT), and lightweight OCR-oriented Chain-of-Thought prompting. On a held-out real-world multilingual OCR benchmark, OCR-SFT improves OCR completeness from 71.3 to 84.6, reduces hallucination rate from 18.3\% to 5.5\%, and improves translation BLEU-1 from 52.3 to 80.2, with substantial hallucination reductions under blur and rotation. Evaluation on public benchmarks further shows gains on OCR-intensive tasks while largely preserving broader multimodal capabilities; ablations show that SFT provides the primary improvement, with prompting offering smaller complementary gains. These results demonstrate that data-centric OCR-aware post-training provides a practical and scalable approach to improving multilingual visual-text grounding in general-purpose MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
