Caliper: 探究LLM中的词汇锚点与因果结构
Caliper: Probing Lexical Anchors versus Causal Structure in LLMs
AI总结：
通过词汇匿名化扰动，揭示大语言模型在因果推理基准上的表现主要依赖词汇模式匹配而非结构因果推理。
摘要：
大语言模型在CLadder等因果推理基准上达到50%至70%的准确率，但尚不清楚这反映的是结构推理还是词汇模式匹配。我们引入Caliper，一种受控扰动方法，在保留每个问题的因果图和概率规范的同时，用占位符标记替换语义变量名。在九个指令微调LLM（从3.8B到671B参数）和三个因果推理基准上，词汇匿名化在本地3.8B-14B模型集上导致稳健的准确率下降，分别为+7.6、+27.0和+11.1个百分点；在跨越2024-2026代际的九个前沿模型上，CRASS和e-CARE上的下降幅度升至+29.6和+18.0个百分点。在40个模型-基准组合中，39个显示出正差距，而在CLadder的伪词子集上，差距缩小了17倍。结构化提示和少样本上下文学习各自缩小了差距，但主要是通过降低较小模型上的P0准确率，而非恢复P1。当前指令微调LLM在零样本评估下，一旦移除词汇锚点，几乎没有证据表明其具备结构因果推理能力。
英文摘要：
Large language models reach 50 to 70% accuracy on causal reasoning benchmarks such as CLadder, but it is unclear whether this reflects structural reasoning or lexical pattern matching. We introduce Caliper, a controlled perturbation that replaces semantic variable names with placeholder tokens while preserving the causal graph and probabilistic specification of each question. Across nine instruction-tuned LLMs from 3.8B to 671B and three causal reasoning benchmarks, lexical anonymization yields robust accuracy drops of +7.6, +27.0, and +11.1 pp on a local 3.8B-14B set, rising to +29.6 and +18.0 pp on CRASS and e-CARE across nine frontier models spanning the 2024-2026 generations. Of 40 engaged model-by-benchmark cells, 39 show a positive gap, and the gap collapses by 17x on CLadder's pseudoword subset. Structured scaffolding and few-shot in-context learning each narrow the gap, but mainly by lowering P0 accuracy on smaller models rather than recovering P1. Current instruction-tuned LLMs, evaluated zero-shot, show little evidence of structural causal reasoning once lexical anchors are removed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
