MechReasoner：定性物理中机制推理的模拟器与基准
MechReasoner: A Simulator and Benchmark for Mechanistic Reasoning in Qualitative Physics
浏览论文内容
中文总结 AI 辅助
本文提出MechReasoner，一个基于合流定性物理的模拟器和基准，用于测试大语言模型在机制推理中的可靠性，实验显示GPT-5.5准确率随机制复杂性增加而显著下降。
中文摘要 AI 辅助
本文介绍了MechReasoner，一个基于合流定性物理的机制性定性模拟器，以及一个用于机制推理的基准。当前的大语言模型（LLMs）生成的机制描述流畅，但并不能可靠地从底层结构和因果约束中推导出来。该基准测试答案是否保留了模拟器许可的模糊性、量化声明、情节图转换证据、修复和轨迹支持判断。其1120个项目由可接受的解释集、组件状态、场景限制、合流约束以及跨18个目录机制和六个任务族的推导步骤确定性生成。每个机制都经过转换器对结构和拓扑的检查，以及针对定量模拟的行为检查。GPT-5.5的准确率随着特定族机制复杂性的增加而下降，从最低复杂性桶（B1）的76.1%降至最高复杂性桶（B4）的38.0%。在控制渲染提示和预期答案长度后，负相关仍然存在。这些结果表明，定性模拟器可以支持用于机制推理的可审计的NLP基准。
英文摘要
This work introduces MechReasoner, a mechanistic qualitative simulator grounded in confluence-based qualitative physics, together with a benchmark for mechanistic inference. Current large language models (LLMs) generate fluent mechanistic descriptions that do not reliably follow from underlying structural and causal constraints. The benchmark tests whether answers preserve simulator-licensed ambiguity, quantified claims, episode-graph transition evidence, repairs, and trace-support judgments. Its 1,120 items are generated deterministically from admissible interpretation sets, component states, scenario restrictions, confluence constraints, and derivation steps across 18 catalog mechanisms and six task families. Each mechanism undergoes converter checks of structure and topology and behavioral checks against quantitative simulations. GPT-5.5 accuracy decreases as family-specific mechanistic complexity increases, from 76.1% in the lowest-complexity bucket (B1) to 38.0% in the highest-complexity bucket (B4). The negative association remains after controls for rendered-prompt and expected-answer length. These results show that qualitative simulators can support auditable NLP benchmarks for mechanistic inference.
发表机构
- Idiap Research Institute（伊迪亚普研究所）
- École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
- University of Manchester（曼彻斯特大学）
机构由 AI 辅助整理，请以论文原文为准。
