大语言模型是否在形式化中作弊？评估逻辑推理中的忠实性
Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning
AI总结：
本文评估了大语言模型在逻辑推理中的忠实性，发现统一生成方法在形式化过程中未表现出系统性作弊行为，但两种阶段流程揭示了两种不同的不忠实性模式。
摘要：
形式验证保证了证明的有效性，但不保证形式化的忠实性。对于自然语言逻辑推理，模型从头构建公理系统而不受库约束，这种有效证明与忠实翻译之间的差距尤为明显。我们研究了前沿模型在生成Lean 4证明时是否利用这一差距，这种行为我们称为形式化作弊。我们评估了GPT-5和DeepSeek-R1在303个一阶逻辑问题上的表现（203个来自FOLIO，100个来自Multi-LogiEval），比较了统一生成方法与分阶段流程（将形式化与证明分离）。尽管编译率在87-99%之间，我们未发现统一生成中存在系统性作弊的证据：模型更倾向于报告失败而非强制生成证明，即使在提示设计鼓励其生成的情况下。然而，可能逃避我们检测的不忠实性仍可能存在。两种阶段流程揭示了两种不同的不忠实性模式：GPT-5在证明生成过程中伪造公理，这种反应性回退可通过跨阶段比较检测；而DeepSeek-R1在形式化过程中误译前提，生成内部一致的输出，完全逃避检测。这些发现表明，高编译率或准确性不应等同于忠实推理。代码和数据可在https://github.com/koreankiwi99/formalization-gaming获取。
英文摘要：
Formal verification guarantees proof validity but not formalization faithfulness. For natural-language logical reasoning, where models construct axiom systems from scratch without library constraints, this gap between valid proofs and faithful translations is especially acute. We investigate whether frontier models exploit this gap when generating Lean 4 proofs, a behavior we term formalization gaming. We evaluate GPT-5 and DeepSeek-R1 on 303 first-order logic problems (203 from FOLIO, 100 from Multi-LogiEval), comparing unified generation against a two-stage pipeline that separates formalization from proving. Despite compilation rates of 87-99%, we find no evidence of systematic gaming in unified generation: models prefer reporting failure over forcing proofs, even under prompting designed to encourage it. However, unfaithfulness that evades our detection signals may still occur. The two-stage pipeline reveals two distinct modes of unfaithfulness: GPT-5 fabricates axioms during proof generation, a reactive fallback detectable via cross-stage comparison, while DeepSeek-R1 mistranslates premises during formalization, producing internally consistent outputs that evade detection entirely. These findings show that high compilation rates or accuracies should not be equated with faithful reasoning. Code and data are available at https://github.com/koreankiwi99/formalization-gaming.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
