CausalT5k: 诊断可信因果推理中的拒绝与失败模式——跨越因果阶梯
CausalT5k: Diagnosing Refusal and Failure Modes in Trustworthy Causal Reasoning Across Causal Rungs
AI总结：
提出CTK基准，通过5,147个案例诊断大语言模型在因果推理中的失败模式，包括因果阶梯、陷阱类型、压力敏感性和拒绝质量等标注，揭示聚合准确率隐藏的缺陷。
摘要：
大型语言模型越来越能生成流畅的因果解释，但它们常常以聚合准确率无法诊断的方式失败：混淆关联与干预、在压力下放弃正确判断、过度拒绝有效主张、或在证据不足时作答。我们引入CTK，一个包含5,147个案例且不断增长的诊断基准，涵盖10个领域和Pearl因果阶梯的所有三个层次。与仅评分的基准不同，CTK通过标注因果阶梯、陷阱类型、压力敏感性、拒绝质量以及效用-安全权衡来揭示模型为何失败。其Sheep/Wolf分类法区分有效因果设计与推理陷阱；配对的neutral/pressure变体通过Bad Flip Rate测量谄媚漂移；Wise Refusal字段测试模型在认可主张前是否识别出缺失信息。CTK暴露了聚合准确率隐藏的失败模式：怀疑陷阱、缩放下的阶梯坍塌、压力诱导漂移、检测-纠正差距以及反事实错误模式。它不规定修正方法，而是为研究因果推理失败概况提供诊断基础。
英文摘要：
Large language models increasingly produce fluent causal explanations, yet they often fail in ways aggregate accuracy cannot diagnose: confusing association with intervention, abandoning correct judgments under pressure, over-refusing valid claims, or answering when evidence is underdetermined. We introduce CTK, a diagnostic benchmark of 5,147 cases and growing, across 10 domains and all three levels of Pearl's Ladder of Causation. Unlike benchmarks that only score correctness, CTK reveals why a model failed by annotating causal rung, trap type, pressure sensitivity, refusal quality, and Utility-Safety tradeoffs. Its Sheep/Wolf taxonomy separates valid causal designs from inferential traps; paired neutral/pressure variants measure sycophantic drift through Bad Flip Rate; and Wise Refusal fields test whether a model identifies the missing information needed before endorsing a claim. CTK exposes failure modes hidden by aggregate accuracy: the Skepticism Trap, Rung Collapse under scaling, pressure-induced drift, Detection-Correction gaps, and counterfactual error modes. Rather than prescribing a correction method, it provides the diagnostic substrate for studying causal-reasoning failure profiles.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
