C2-Faith: 为思维链推理中的因果和覆盖忠实性基准测试LLM评判者
C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning
AI总结：
提出C2-Faith基准，通过因果和覆盖两个维度评估LLM评判者对思维链推理过程忠实性的判断能力，发现模型在错误定位和覆盖评分上存在显著不足。
摘要：
大型语言模型（LLM）越来越多地被用作思维链（CoT）推理的评判者，但目前尚不清楚它们能否可靠地评估过程忠实性，而不仅仅是答案的合理性。我们引入了C2-Faith，这是一个基于PRM800K构建的基准，明确将忠实性分解为两个互补维度：因果性（每一步是否逻辑上源自先前上下文）和覆盖性（是否包含必要的中间推理）。通过受控扰动，我们构建了具有已知因果错误位置的示例，将单个步骤替换为逻辑不一致的变体，并以不同速率进行受控覆盖删除，从而能够直接根据参考标签进行测量。我们评估了三个前沿的LLM评判者在三项任务上的表现：二元因果检测、因果步骤定位和覆盖评分。我们的结果表明，评判者的可靠性高度依赖于任务，没有单一模型在所有设置中占主导地位。虽然模型通常能检测到错误存在，但它们难以准确定位错误，这表明检测与归因之间存在显著差距。此外，所有评判者都系统性地高估了推理完整性，即使中间推理的很大部分缺失，也会给出高覆盖分数。这些发现揭示了LLM评判者在过程级评估中的根本局限性，并强调了在使用LLM评估推理质量时需要更可靠和校准的方法。
英文摘要：
Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning, yet it remains unclear whether they can reliably assess process faithfulness rather than merely answer plausibility. We introduce C2-Faith, a benchmark built from PRM800K that explicitly decomposes faithfulness into two complementary dimensions: causality (whether each step logically follows from prior context) and coverage (whether essential intermediate inferences are present). Using controlled perturbations, we construct examples with known causal error positions by replacing a single step with a logically inconsistent variant, and with controlled coverage deletions at varying rates, enabling direct measurement against reference labels. We evaluate three frontier LLM judges across three tasks: binary causal detection, causal step localization, and coverage scoring. Our results reveal that judge reliability is highly task-dependent, with no single model dominating across settings. While models often detect that an error exists, they struggle to accurately localize it, indicating a substantial gap between detection and attribution. Moreover, all judges systematically overestimate reasoning completeness, assigning high coverage scores even when substantial portions of intermediate reasoning are missing. These findings expose fundamental limitations of LLM judges in process-level evaluation and highlight the need for more reliable and calibrated methods when using LLMs to assess reasoning quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
