通过符号验证揭示LLM因果推理中的隐藏正确性
Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification
AI总结：
本文提出DoVerifier，通过符号验证揭示LLM因果推理中的隐藏正确性，提升因果推理评估的严谨性与信息量。
摘要：
大型语言模型（LLMs）正越来越多地应用于涉及因果推理的任务。然而，当前的基准测试往往依赖于字符串匹配或表面级指标，这些指标无法捕捉到模型输出在因果推理语义下的形式有效性。为了解决这一问题，我们提出了DoVerifier，一种简单的符号验证器，它检查LLM生成的因果表达式是否可以通过给定的因果图和do- calculus及概率论中的规则推导出来。这使我们能够恢复那些因因果语义表面差异而被标记为错误的因果查询的正确答案。我们在合成数据和因果问答基准测试中的评估表明，DoVerifier更准确地捕捉了因果推理轨迹的语义正确性，提供了一种更严谨且信息丰富的方式来评估LLM在因果推理上的表现。
英文摘要：
Large language models (LLMs) are increasingly being applied to tasks that involve causal reasoning. However, current benchmarks often rely on string matching or surface-level metrics that do not capture whether the output of a model is formally valid under the semantics of causal reasoning. To address this, we propose DoVerifier, a simple symbolic verifier that checks whether LLM-generated causal expressions are derivable from a given causal graph using rules from do-calculus and probability theory. This allows us to recover correct answers to causal queries that would otherwise be marked incorrect due to superficial differences in their causal semantics. Our evaluations on synthetic data and causal QA benchmarks show that DoVerifier more accurately captures semantic correctness of causal reasoning traces, offering a more rigorous and informative way to evaluate LLMs on causal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
