arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.21709cs.CL
收藏

为何存在注意力模式:一种统一的时间视角分析

Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(脑启发智能感知与认知教育部重点实验室,中国科学技术大学);Huawei Technologies Co., Ltd.(华为技术有限公司);College of Intelligence and Computing, Tianjin University(智能与计算学院,天津大学)

Qingyue Yang, Jie Wang, Xing Li, Yinqi Bai, Xialiang Tong, Huiling Zhen, Jianye Hao, Mingxuan Yuan, Bin Li

AI总结:

本文提出TAPPA框架,从时间连续性角度分析注意力模式,揭示其可预测性与查询自相似性关系,并通过KV缓存压缩和剪枝任务验证其有效性。

摘要:

注意力模式在大型语言模型(LLMs)的训练和推理中起着至关重要的作用。先前的工作已经识别出个体模式,如检索头、汇头和对角线轨迹,但这些观察结果仍然零散且缺乏统一的解释。为了填补这一空白,我们引入了时间注意力模式可预测性分析(TAPPA),一种统一的框架,通过从时间连续性角度分析其底层数学公式来解释各种注意力模式。TAPPA不仅加深了对注意力行为的理解,还指导了推理加速方法。具体而言,TAPPA将注意力模式分为可预测模式和不可预测模式。我们的分析进一步揭示,这种区分可以由时间维度上查询自相似性的程度来解释。针对可预测模式,我们进一步通过查询、键和旋转位置嵌入(RoPE)的联合效应,对三种代表性案例进行了详细的数学分析。我们通过将其见解应用于KV缓存压缩和LLM剪枝任务来验证TAPPA。在这些任务中,一个由TAPPA启发的简单度量指标在基线方法上持续改进性能。代码可在https://github.com/MIRALab-USTC/LLM-TAPPA上获得。

英文摘要:

Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce \textbf{Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations} from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑