ICE：基于统计基础的LLM干预一致解释评估
ICE: Intervention-Consistent Explanation Evaluation with Statistical Grounding for LLMs
AI总结：
本文提出ICE框架，通过多干预操作比较解释与随机基线，发现解释一致性依赖于干预操作，且不同语言模型在不同任务上的表现存在显著差异，揭示了模型与语言之间的复杂交互。
摘要：
评估解释是否真实反映模型推理仍是一个开放性问题。现有基准测试使用单一干预且无统计检验，无法区分真实一致性与偶然表现。本文引入ICE（干预一致解释）框架，通过随机化测试在多个干预操作下比较解释与匹配随机基线，得到具有置信区间的胜率。对7个LLM在4个英语任务、6个非英语语言和2种归因方法上进行评估，发现一致性依赖于干预操作：干预差距可达44个百分点，删除通常在短文本中夸大估计，但长文本中模式反转，表明一致性应比较不同干预操作而非单一评分。随机基线揭示三分之一的配置存在反一致性，一致性与人类合理性无相关性（|r| < 0.04）。多语言评估揭示了无法仅通过分词解释的模型-语言交互。我们发布了ICE框架和ICEBench基准。
英文摘要：
Evaluating whether explanations faithfully reflect a model's reasoning remains an open problem. Existing benchmarks use single interventions without statistical testing, making it impossible to distinguish genuine faithfulness from chance-level performance. We show that faithfulness is not a fixed property but an operator-dependent quantity that changes with the intervention method used to measure it. We introduce ICE (Intervention-Consistent Explanation), a framework that evaluates explanations against random baselines of equal size under multiple operators. Evaluating 7 LLMs across 4 tasks with deletion and retrieval infill operators, we find that switching operators crosses the positive-evidence threshold in 18% of configurations (5 of 28 attention comparisons), with gaps reaching 44 percentage points. Randomized baselines detect anti-faithfulness (explanations worse than random) in nearly one-third of English deletion configurations, invisible without random comparisons. These patterns persist across 6 non-English languages and 2 attribution methods. The methodology generalizes to step-level chain-of-thought evaluation, where preliminary results on 3 frontier models suggest that high accuracy does not imply faithful reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
