OCR元推理基准：评估多模态大语言模型在富文本图像理解中的元推理能力
OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding
浏览论文内容
中文总结 AI 辅助
本文提出OCR-MetaReasoning基准，将演绎、归纳、溯因作为不同推理方向，分离答案正确性与推理合规性，实验发现现有MLLMs的OCR元推理能力远未饱和。
中文摘要 AI 辅助
富文本图像理解要求多模态大语言模型（MLLMs）组织基于光学字符识别（OCR）的证据，涵盖文字、布局、字段、图表及视觉对应关系。现有评估常将提取与推理混淆，且极少测试模型是否遵循所需推理方向：应用可见规则、抽象隐藏规律或恢复缺失前提。本文提出OCR-MetaReasoning，这是一个受控单图像基准，将演绎、归纳、溯因视为不同方向，并将最终答案正确性与推理过程合规性分离。该基准包含1500个经验证的样本，构成3×5平衡分类体系，涵盖三种推理类型与五个OCR对象类别，同时提供参考推理步骤、自动答案评分、元推理宏观评分（MRMS）及推理过程合规评分（RPCS）。对代表性闭源与开源MLLMs的实验显示，基于OCR的元推理远未达饱和状态：模型在可见规则应用及布局敏感推理上表现不佳，且在精确匹配评估下，推理过程合规的理由可能伴随错误的最终答案。代码可在此httpsURL获取。
英文摘要
Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across words, layout, fields, charts, and visual correspondences. Existing evaluations often conflate extraction with reasoning and rarely test whether models follow the required reasoning direction: applying visible rules, abstracting hidden regularities, or recovering missing premises. We introduce OCR-MetaReasoning, a controlled single-image benchmark that treats deduction, induction, and abduction as distinct directions and separates final-answer correctness from reasoning-process compliance. The benchmark contains 1,500 verified samples in a balanced \(3\times5\) taxonomy crossing three reasoning types with five OCR-object categories, along with reference reasoning steps, automatic answer scoring, the Meta-Reasoning Macro Score (MRMS), and the Reasoning Process Compliance Score (RPCS). Experiments with representative closed-source and open-source MLLMs show that OCR-grounded meta-reasoning remains far from saturated: models struggle with visible-rule application and layout-sensitive inference, while process-compliant rationales can accompany incorrect final answers under exact-match evaluation. The code is available at https://github.com/gengxuli/OCR-MetaReasoning.
发表机构
- School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）
- International Center of Future Science, Jilin University（吉林大学未来科学国际中心）
机构由 AI 辅助整理，请以论文原文为准。
