AI 中文总结
针对机械工程多图像多跳推理评估缺失，提出源自真实论文的 MechReason 基准，含 1.2 万问答对和 2.1 万视觉材料，覆盖四维八类任务，最先进模型准确率仅 62.89%。
AI 中文摘要
尽管通用视觉问答和跨模态理解取得了显著进展，多模态大语言模型在机械工程领域内复杂推理的评估方面仍存在明显空白。现有基准主要关注诸如图纸识别、CAD 解释或单一图表查询等基础任务，未能评估模型在面对真实、复杂的机械问题时，能否整合多张图像、文本条件、物理原理和工程约束，执行多步推理。为解决这一问题，我们提出了 MechReason，一个源自真实机械工程论文的基准，包含 12,000 个带有显式推理链标注的问答对，以及涵盖九种证据类型的 21,000 份视觉材料，包括统计图表、参数表、工程图纸、显微图像、仿真图像、系统架构、真实机械场景照片、CAD 模型图像和制造流程图。MechReason 覆盖了四个推理维度（解释、预测、设计和诊断）下的八种任务类型。我们设计了一个四阶段构建流程：首先提取核心工程主张，并将其支持证据分解为前提、推理过程、结论和佐证证据；然后通过掩盖后验验证信息生成防止捷径的问题；最后应用多模态质量验证以确保任务质量和多跳特性。大量实验结果表明，MechReason 极具挑战性，即使最先进的模型也仅达到 62.89% 的准确率。
英文摘要
Despite significant progress in general visual question answering and cross-modal understanding, multimodal large language models still face a pronounced gap in evaluation for complex reasoning within the mechanical engineering domain. Existing benchmarks predominantly focus on rudimentary tasks such as drawing recognition, CAD interpretation, or single-chart querying, falling short of assessing whether models can integrate multiple images, textual conditions, physical principles, and engineering constraints to perform multi-step reasoning when confronted with authentic, intricate mechanical problems. To address this, we introduce MechReason, a benchmark derived from real mechanical engineering papers, comprising 12k question-answer pairs with explicit reasoning-chain annotations and 21k visual materials spanning nine evidence types, including statistical charts, parameter tables, engineering drawings, microscopic images, simulation images, system architectures, real mechanical scene photos, CAD model images and manufacturing flowcharts. MechReason covers eight task types across four reasoning dimensions: explanation, prediction, design, and diagnosis. We devise a four-stage construction pipeline: we first extract core engineering claims and decompose their supporting evidence into premises, reasoning processes, conclusions, and corroborative evidence; we then generate shortcut-preventing questions by masking posterior verification information; finally, we apply multimodal quality validation to ensure task quality and multi-hop nature. Extensive experimental results demonstrate that MechReason is highly challenging, with even the most advanced models achieving only 62.89\% accuracy.
