冗长性权衡与规模对LLM自我解释忠实度的影响
Verbosity Tradeoffs and the Impact of Scale on the Faithfulness of LLM Self-Explanations
AI总结：
本文分析13个模型家族共75个模型的反事实忠实度，提出phi-CCT和F-AUROC两个新指标，发现更大更强的模型在所有指标上更忠实。
摘要：
当被要求解释其决策时，LLM通常能给出对人类来说听起来合理的解释。但这些解释是否忠实，即它们是否传达了实际负责决策的因素？在这项工作中，我们分析了来自13个模型家族的75个模型的反事实忠实度。我们分析了简洁性与全面性之间的权衡，相关性忠实度指标如何评估这种权衡，以及指标在多大程度上可以被操纵。这一分析推动了两个新指标：phi-CCT，它是相关性反事实测试（CCT）的简化变体，避免了令牌概率的需求，同时解释了原始测试的大部分方差；以及F-AUROC，它消除了对不平衡干预分布的敏感性，并捕捉了模型生成不同细节水平解释的能力。我们的发现揭示了一个清晰的缩放趋势：更大更强的模型在我们考虑的所有指标上始终更忠实。我们的代码可在以下网址获取：https://this https URL。
英文摘要：
When asked to explain their decisions, LLMs can often give explanations which sound plausible to humans. But are these explanations faithful, i.e. do they convey the factors actually responsible for the decision? In this work, we analyse counterfactual faithfulness across 75 models from 13 families. We analyze the tradeoff between conciseness and comprehensiveness, how correlational faithfulness metrics assess this tradeoff, and the extent to which metrics can be gamed. This analysis motivates two new metrics: the phi-CCT, a simplified variant of the Correlational Counterfactual Test (CCT) which avoids the need for token probabilities while explaining most of the variance of the original test; and F-AUROC, which eliminates sensitivity to imbalanced intervention distributions and captures a model's ability to produce explanations with different levels of detail. Our findings reveal a clear scaling trend: larger and more capable models are consistently more faithful on all metrics we consider. Our code is available at https://github.com/google-deepmind/corr_faith.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
