PlotChain: 多模态大语言模型在工程图表阅读中的确定性检查点评估
PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading
AI总结：
PlotChain提出了一种确定性的基于生成器的评估基准，用于评估多模态大语言模型在工程图表阅读中的性能，通过检查点实现子技能诊断和故障定位，展示了不同模型在图表阅读任务中的表现差异。
摘要：
我们提出了PlotChain，一种确定性的、基于生成器的基准，用于评估多模态大语言模型（MLLMs）在工程图表阅读中的性能，即从经典图表（如Bode/FFT、阶跃响应、应力-应变、泵曲线）中恢复定量值，而非仅依赖OCR提取或自由形式描述。PlotChain包含15种图表家族，共450张渲染图表（每类30张），其中每个项目均从已知参数生成，并配以直接从生成过程计算出的精确地面真实值。核心贡献是基于检查点的诊断评估：除了最终目标外，每个项目还包括中间'cp_'字段，用于隔离子技能（例如读取截止频率或峰值幅度），并实现对图表家族内部的故障定位。我们采用标准化、确定性的协议（温度=0和严格的JSON-only数值输出模式）评估四个最先进的MLLMs，并使用每字段容忍度评分预测，以反映人类图表阅读的精度。在'plotread'容忍策略下，顶级模型在字段级别通过率分别为80.42%（Gemini 2.5 Pro）、79.84%（GPT-4.1）和78.21%（Claude Sonnet 4.5），而GPT-4o则为61.59%。尽管在许多家族中表现强劲，但频域任务仍脆弱：带通响应仍低（<=23%），FFT谱仍具挑战性。我们发布生成器、数据集、原始模型输出、评分代码和清单，并附带校验和，以支持完全可重复的运行和在替代容忍策略下的回顾性重新评分。
英文摘要：
We present PlotChain, a deterministic, generator-based benchmark for evaluating multimodal large language models (MLLMs) on engineering plot reading-recovering quantitative values from classic plots (e.g., Bode/FFT, step response, stress-strain, pump curves) rather than OCR-only extraction or free-form captioning. PlotChain contains 15 plot families with 450 rendered plots (30 per family), where every item is produced from known parameters and paired with exact ground truth computed directly from the generating process. A central contribution is checkpoint-based diagnostic evaluation: in addition to final targets, each item includes intermediate 'cp_' fields that isolate sub-skills (e.g., reading cutoff frequency or peak magnitude) and enable failure localization within a plot family. We evaluate four state-of-the-art MLLMs under a standardized, deterministic protocol (temperature = 0 and a strict JSON-only numeric output schema) and score predictions using per-field tolerances designed to reflect human plot-reading precision. Under the 'plotread' tolerance policy, the top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%. Despite strong performance on many families, frequency-domain tasks remain brittle: bandpass response stays low (<= 23%), and FFT spectrum remains challenging. We release the generator, dataset, raw model outputs, scoring code, and manifests with checksums to support fully reproducible runs and retrospective rescoring under alternative tolerance policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
