通过对比协方差分解查询-键特征交互
Decomposing Query-Key Feature Interactions Using Contrastive Covariances
AI总结：
本文提出通过对比协方差分解查询-键特征交互，以揭示Transformer中注意力机制的可解释性。
摘要：
尽管注意力头在Transformer中扮演着核心角色，但我们缺乏理解模型为何关注特定标记的工具。为解决这一问题，我们研究了查询-键（QK）空间——查询和键之间的双线性联合嵌入空间。我们提出了一种对比协方差方法，将QK空间分解为低秩、人类可解释的组件。当键和查询中的特征在这些低秩子空间中对齐时，会产生高注意力分数。我们首先在简化设置中从分析和经验上研究了我们的方法。然后我们将该方法应用于大型语言模型，以识别人类可解释的QK子空间用于类别语义特征和绑定特征。最后，我们展示了如何将注意力分数归因于我们识别出的特征。
英文摘要：
Despite the central role of attention heads in Transformers, we lack tools to understand why a model attends to a particular token. To address this, we study the query-key (QK) space -- the bilinear joint embedding space between queries and keys. We present a contrastive covariance method to decompose the QK space into low-rank, human-interpretable components. It is when features in keys and queries align in these low-rank subspaces that high attention scores are produced. We first study our method both analytically and empirically in a simplified setting. We then apply our method to large language models to identify human-interpretable QK subspaces for categorical semantic features and binding features. Finally, we demonstrate how attention scores can be attributed to our identified features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
