从答案到状态：大语言模型中化学推理的可验证过程级评估
From Answers to States: Verifiable Process-Level Evaluation of Chemical Reasoning in Large Language Models
AI总结：
提出ChemCoTBench-V2基准，通过确定性规则和参考轨迹验证结构化化学推理步骤，揭示模型在最终答案正确性与推理状态一致性之间的差距。
摘要：
大语言模型越来越多地被用作化学助手，然而大多数化学基准仍然只对最终答案评分。这掩盖了一个关键的失败模式：模型可能输出正确的分子、产物或选项，但其推理过程违反了化学逻辑。现有的过程级评估器难以扩展，因为LLM评判者和人工步骤级过程注释成本高、不一致且容易产生幻觉。我们引入了ChemCoTBench-V2，一个规则可验证的诊断基准，用于对结构化、可验证的化学推理轨迹进行低成本、可审计的评估。它涵盖分子理解、分子编辑、分子优化和反应预测，包含18个报告任务中的5620个评估样本。模型必须在专家设计的模板中暴露关键中间步骤，这些步骤通过确定性化学规则进行检查，对于封闭答案任务，还使用参考轨迹而非另一个LLM评判者。开放式的分子优化通过预言机可验证的状态约束而非严格的轨迹匹配进行评估。该基准报告三个独立的信号：最终答案正确性、模板遵循度和基于专家精炼中间步骤的逐步骤验证器正确性。对前沿模型的实验揭示了最终答案成功与结构化推理状态一致性之间的持续差距：模型通常遵循要求的格式但未能通过化学步骤检查，或者正确回答但支持性推理薄弱。ChemCoTBench-V2支持细粒度模型比较，并识别轨迹首次违反验证器的具体步骤。
英文摘要：
Large language models are increasingly used as chemistry assistants, yet most chemistry benchmarks still score only final answers. This masks a critical failure mode: a model may output the correct molecule, product, or option while its reasoning violates chemical logic. Existing process-level evaluators are hard to scale because LLM judges and human step-level process annotation are costly, inconsistent, and vulnerable to hallucination. We introduce ChemCoTBench-V2, a rule-verifiable diagnostic benchmark for low-cost, auditable evaluation of structured, verifier-addressable chemical reasoning traces. It spans molecular understanding, molecule editing, molecular optimization, and reaction prediction, with 5,620 evaluation samples across 18 reporting tasks. Models must expose key intermediate steps in expert-designed templates, and those steps are checked with deterministic chemistry rules and, for closed-answer tasks, reference traces rather than another LLM judge. Open-ended molecular optimization is evaluated with oracle-verifiable state constraints rather than strict trace matching. The benchmark reports three separate signals: final-answer correctness, template adherence, and step-wise verifier correctness over expert-refined intermediate commitments. Experiments on frontier models reveal a persistent gap between final-answer success and structured-reasoning-state consistency: models often follow the requested format while failing chemical-step checks, or answer correctly with weak supporting reasoning. ChemCoTBench-V2 enables fine-grained model comparison and identifies the concrete step at which the trace first violates the verifier.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
