OCR-Reasoning基准：揭示MLLMs在复杂文本丰富图像推理中的真实能力
OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning
AI总结：
提出OCR-Reasoning基准，包含1069个人工标注样本，覆盖6种核心推理能力和18个实际推理任务，通过双标注（最终答案和逐步推理过程）评估多模态大语言模型在文本丰富图像推理中的能力，发现最先进模型准确率均低于50%。
摘要：
近期多模态慢思考系统在各种视觉推理任务中表现出色。然而，由于缺乏专门且系统的基准，它们在文本丰富图像推理任务中的能力仍未得到充分研究。为填补这一空白，我们提出了OCR-Reasoning，一个新颖的基准，旨在系统评估多模态大语言模型在文本丰富图像推理任务上的表现。具体而言，OCR-Reasoning包含1069个人工标注的示例，涵盖文本丰富视觉场景中的6种核心推理能力和18个实际推理任务。与仅提供最终答案的现有文本丰富图像理解基准不同，本基准额外提供了详细的逐步推理过程。这种双标注使得能够同时评估模型的最终答案和推理过程，从而全面评估文本丰富推理能力。利用该基准，我们对最新的多模态大语言模型进行了全面评估。结果表明，即使是最先进的多模态大语言模型在文本丰富图像推理任务中也面临巨大困难，在我们的基准上没有一个模型的准确率超过50%，这表明文本丰富图像推理的挑战是一个亟待解决的问题。基准和评估脚本可在https://github.com/SCUT-DLVCLab/OCR-Reasoning获取。
英文摘要：
Recent advancements in multimodal slow-thinking systems have demonstrated remarkable performance across various visual reasoning tasks. However, their capabilities in text-rich image reasoning tasks remain understudied due to the absence of a dedicated and systematic benchmark. To address this gap, we propose OCR-Reasoning, a novel benchmark designed to systematically assess Multimodal Large Language Models on text-rich image reasoning tasks. Specifically, OCR-Reasoning comprises 1,069 human-annotated examples spanning 6 core reasoning abilities and 18 practical reasoning tasks in text-rich visual scenarios. Unlike existing text-rich image understanding benchmarks that only provide a final answer, this benchmark additionally provides a detailed step-by-step reasoning process. This dual annotation enables the evaluation of both the models' final answers and their reasoning processes, thereby offering a holistic assessment of text-rich reasoning capabilities. By leveraging this benchmark, we conducted a comprehensive evaluation of the latest MLLMs. Our results demonstrate that even the most advanced MLLMs exhibit substantial difficulties in text-rich image reasoning tasks, with none achieving an accuracy above 50\% on our benchmark, indicating that the challenges of text-rich image reasoning are an urgent issue to be addressed. The benchmark and evaluation scripts are available at https://github.com/SCUT-DLVCLab/OCR-Reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
