Hessian-Enhanced Token Attribution (HETA): 解释自回归语言模型
Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs
AI总结：
HETA提出了一种针对解码器-only语言模型的新型解释框架，结合语义过渡向量、Hessian敏感度评分和KL散度，提升上下文感知和因果解释能力，通过基准数据集验证其在生成任务中的优越性。
摘要：
Attribution方法旨在通过量化输入token对生成输出的贡献来解释语言模型的预测。然而，现有技术多针对编码器架构，依赖线性近似，无法捕捉解码器-only模型中自回归生成的因果和语义复杂性。为此，我们提出Hessian-Enhanced Token Attribution (HETA)，一种专为解码器-only语言模型设计的新型解释框架。HETA结合三个互补组件：语义过渡向量捕捉token到token的层间影响，Hessian基于敏感度评分建模二次效应，KL散度衡量token被屏蔽时的信息损失。这种统一设计产生上下文感知、因果忠实且语义扎根的解释。此外，我们引入了一个经过精心挑选的基准数据集，系统评估生成场景中的解释质量。在多个模型和数据集上的实证评估表明，HETA在解释忠实性和与人类注释的一致性上始终优于现有方法，为自回归语言模型的可解释性树立了新标准。
英文摘要：
Attribution methods seek to explain language model predictions by quantifying the contribution of input tokens to generated outputs. However, most existing techniques are designed for encoder-based architectures and rely on linear approximations that fail to capture the causal and semantic complexities of autoregressive generation in decoder-only models. To address these limitations, we propose Hessian-Enhanced Token Attribution (HETA), a novel attribution framework tailored for decoder-only language models. HETA combines three complementary components: a semantic transition vector that captures token-to-token influence across layers, Hessian-based sensitivity scores that model second-order effects, and KL divergence to measure information loss when tokens are masked. This unified design produces context-aware, causally faithful, and semantically grounded attributions. Additionally, we introduce a curated benchmark dataset for systematically evaluating attribution quality in generative settings. Empirical evaluations across multiple models and datasets demonstrate that HETA consistently outperforms existing methods in attribution faithfulness and alignment with human annotations, establishing a new standard for interpretability in autoregressive language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
