arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13524cs.LGcs.AI
收藏

注意力头的奇异向量与特征对齐

Singular Vectors of Attention Heads Align with Features

Department of Computer Science, Boston University, Boston, USA;Faculty of Computing \& Data Sciences, Boston University, Boston, USA

Gabriel Franco, Carson Loughridge, Mark Crovella

AI总结:

本文通过理论分析和实验验证,解释了注意力头奇异向量与特征表示对齐的原因和条件,并提出了稀疏注意力分解作为对齐的可检验预测。

摘要:

识别语言模型中的特征表示是机械可解释性的核心任务。最近的一些研究观察到,在某些情况下,可以从注意力矩阵的奇异向量中推断出特征表示。然而,这一现象缺乏合理的解释。本文探讨了这个问题:为什么以及何时奇异向量与特征对齐?首先,我们证明在可以直接观察特征的模型中,奇异向量与特征稳健地对齐。然后,我们从理论上表明,这种对齐在多种条件下是预期的。最后,我们提出如何在特征表示不可直接观察的真实模型中操作性地识别对齐。我们将稀疏注意力分解确定为对齐的一个可检验预测,并展示证据表明它在真实模型中以与预测一致的方式出现。这些结果共同表明,奇异向量与特征的对齐可以作为语言模型中特征识别的合理且有理论依据的基础。

英文摘要:

Identifying feature representations in language models is a central task in mechanistic interpretability. Several recent studies have made the observation that feature representations can be inferred in some cases from singular vectors of attention matrices. However, sound justification for this phenomenon is lacking. In this paper we address that question, asking: why and when do singular vectors align with features? First, we demonstrate that singular vectors robustly align with features in a model where features can be directly observed. We then show theoretically that such alignment is expected under a range of conditions. We close by asking how, operationally, alignment may be recognized in real models where feature representations are not directly observable. We identify sparse attention decomposition as a testable prediction of alignment, and show evidence that it emerges in real models in a manner consistent with predictions. Together these results suggest that alignment of singular vectors with features can be a sound and theoretically justified basis for feature identification in language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑