衡量忠实性取决于如何测量：分类器敏感性在LLM链式推理评估中的应用
Measuring Faithfulness Depends on How You Measure: Classifier Sensitivity in LLM Chain-of-Thought Evaluation
AI总结：
本文通过不同分类器对10276个推理轨迹的分析，揭示了模型忠实性评估中分类器选择对结果的影响，指出不同方法导致的忠实性差异显著，需采用多种方法进行评估。
摘要：
近期关于链式推理（CoT）忠实性的研究报告单一聚合数值（例如，DeepSeek-R1承认提示39%的时间），暗示忠实性是模型的客观可测量属性。本文提供证据表明并非如此。三个分类器（仅正则表达式检测器、正则表达式加LLM流程和Claude Sonnet 4判断器）被应用于12个开放权重模型的10276个受影推理轨迹，这些模型涵盖9个家族和7B至1T参数。在相同数据上，这些分类器的忠实率分别为74.4%、82.6%和69.7%。每模型差距范围从2.6到30.6个百分点；所有配对的McNemar检验均显著（p < 0.001）。分歧是系统性的：Cohen's kappa从0.06（轻微）到0.42（中等），分别适用于奉承提示和评分提示，且不对称性明显：对于奉承提示，883个案例被流程分类为忠实但被Sonnet判断器分类为不忠实，而仅有2个案例相反。分类器选择也可以逆转模型排名：Qwen3.5-27B在流程下排名第一但在Sonnet下排名第七；OLMo-3.1-32B从第九名变为第三名。不同分类器对忠实性的操作化在不同严格程度（词汇提及与知识依赖）层面，导致相同行为的测量差异。这些结果表明，已发表的忠实性数值不能在不同分类器的跨研究中进行有意义比较，未来评估应报告多种分类方法下的敏感性范围。
英文摘要：
Recent work on chain-of-thought (CoT) faithfulness reports single aggregate numbers (e.g., DeepSeek-R1 acknowledges hints 39% of the time), implying that faithfulness is an objective, measurable property of a model. This paper provides evidence that it is not. Three classifiers (a regex-only detector, a regex-plus-LLM pipeline, and a Claude Sonnet 4 judge) are applied to 10,276 influenced reasoning traces from 12 open-weight models spanning 9 families and 7B to 1T parameters. On identical data, these classifiers produce faithfulness rates of 74.4%, 82.6%, and 69.7%. Per-model gaps range from 2.6 to 30.6 percentage points; all pairwise McNemar tests are significant (p < 0.001). The disagreements are systematic: Cohen's kappa ranges from 0.06 ("slight") for sycophancy hints to 0.42 ("moderate") for grader hints, and the asymmetry is pronounced: for sycophancy, 883 cases are classified as faithful by the pipeline but unfaithful by the Sonnet judge, while only 2 go the other direction. Classifier choice can also reverse model rankings: Qwen3.5-27B ranks 1st under the pipeline but 7th under Sonnet; OLMo-3.1-32B moves from 9th to 3rd. Different classifiers operationalize faithfulness at different levels of stringency (lexical mention versus epistemic dependence), yielding divergent measurements on the same behavior. These results indicate that published faithfulness numbers cannot be meaningfully compared across studies using different classifiers, and that future evaluations should report sensitivity ranges across multiple classification methodologies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
