OrthoRank：基于汇聚令牌正交性的令牌选择以实现高效大语言模型推理
OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
AI总结：
本文提出 OrthoRank 动态令牌选择方法，利用令牌向汇聚令牌移动速度定义重要性，在保持效率的同时提升大语言模型推理性能。
摘要：
注意力机制是大语言模型（LLMs）成功的核心，使其能够捕捉复杂的令牌依赖关系并隐式地为每个令牌分配重要性。近期研究揭示了汇聚令牌（sink token）的存在，尽管其语义作用有限，却获得了不成比例的高注意力。在本文中，我们首先扩展了汇聚令牌与其他令牌之间的关系，超越注意力层面，考虑层深度，探索它们在隐藏状态中的相似性。我们观察到，随着层数加深，汇聚令牌的归一化隐藏状态与其他令牌的归一化隐藏状态之间的余弦相似度增加，并且汇聚令牌的归一化隐藏状态变化极小。这表明其他令牌在各层中持续朝汇聚令牌方向移动。接下来，我们提出了一种动态令牌选择方法，称为 OrthoRank，利用这些发现来选择重要令牌。具体而言，在某一层中，我们通过令牌向汇聚令牌移动的速度来定义令牌重要性。这被转化为与汇聚令牌的正交性，意味着与汇聚令牌更正交的令牌被赋予更高的重要性。最后，通过大量实验，我们证明了与相同稀疏率下的层剪枝方法相比，我们的方法在可比吞吐量下具有更低的困惑度和更高的零样本准确率，同时在 LongBench 上也取得了更优的性能。
英文摘要：
Attention mechanisms are central to the success of large language models (LLMs), enabling them to capture intricate token dependencies and implicitly assign importance to each token. Recent studies have revealed the sink token, which receives disproportionately high attention despite their limited semantic role. In this paper, we first expand the relationship between the sink token and other tokens, moving beyond attention to explore their similarity in hidden states, considering the layer depth. We observe that as the layers get deeper, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink token exhibit negligible changes. These imply that other tokens consistently are directed toward the sink token throughout the layers. Next, we propose a dynamic token selection method, called OrthoRank, using these findings to select important tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink token are assigned greater importance. Finally, through extensive experiments, we demonstrated that our method results in lower perplexity and higher zero-shot accuracy compared to layer pruning methods at the same sparsity ratio with comparable throughput, while also achieving superior performance on LongBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
