PHREEQC-MCQ-200：用于工具增强型科学模拟智能体的诊断基准
PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents
AI总结：
提出PHREEQC-MCQ-200基准，评估工具增强型智能体在地球化学模拟中的表现，发现模拟器访问显著提升准确率但存在非单调性，输出访问协议影响性能。
摘要：
大型语言模型智能体越来越多地连接到科学软件，但目前尚不清楚工具访问何时使科学计算更可靠而非仅仅更复杂。我们引入了PHREEQC-MCQ-200，一个用于评估确定性水-地球化学模拟中工具增强型智能体的基准。该基准包含200个多项选择题，源自21个经过验证的PHREEQC场景，要求智能体构建模拟器输入、执行PHREEQC、检查结构化输出并提交最终答案。在多个前沿和中端模型系列中，模拟器访问显著提高了总体准确率，证实了接地执行对于许多科学计算任务是必要的。然而，这种提升并非单调：工具增强型智能体也会丢失它们在没有工具时正确回答的项目，揭示了仅平均准确率无法掩盖的退化。我们进一步表明，输出访问协议很重要。目录界面可以降低令牌成本，同时保持或提高较强模型的准确率，但会降低无法可靠导航结构化模拟器输出的中端模型的性能。因此，PHREEQC-MCQ-200将科学工具使用视为一个端到端的诊断问题，而非简单的工具调用能力。我们认为，对科学智能体的评估不仅应报告准确率，还应报告项目级保留率、输出访问敏感性、轨迹失败以及计算链断裂的位置。
英文摘要：
Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex. We introduce PHREEQC-MCQ-200, a benchmark for evaluating tool-augmented agents on deterministic aqueous-geochemistry simulations. The benchmark contains 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Across multiple frontier and mid-tier model families, simulator access substantially improves aggregate accuracy, confirming that grounded execution is necessary for many scientific-computation tasks. However, the gains are not monotonic: tool-augmented agents also lose items they answered correctly without tools, revealing regressions that average accuracy alone hides. We further show that output-access protocol matters. A table-of-contents interface can reduce token cost while preserving or improving accuracy for stronger models, but it degrades performance for mid-tier models that cannot reliably navigate structured simulator outputs. PHREEQC-MCQ-200 therefore frames scientific tool use as an end-to-end diagnostic problem rather than a simple tool-calling capability. We argue that evaluations of scientific agents should report not only accuracy, but also item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
