通过稀疏性诱导的正则化训练实现注意力凝聚
Attention Condensation via Sparsity Induced Regularized Training
AI总结:
针对长上下文中自注意力推理开销问题,本文扩展注意力稀疏性理论框架,提出通过限制注意力矩阵顶部元素数量的定制正则化损失进行训练,并在GPT-2上验证其可兼顾稀疏性与依赖建模能力。
摘要:
随着上下文窗口不断扩大,自注意力在transformer推理时间中的占比日益增加。因此,在尽可能降低性能下降的同时加速注意力计算,对于大语言模型(LLM)的高效部署至关重要。本研究扩展了LLM中注意力稀疏性的理论框架。我们设计了一种定制化损失函数,通过限制注意力矩阵中顶部元素的数量来强制实现稀疏性。我们使用GPT-2开展了一组初步评估,以证明该稀疏化方法的有效性。采用所提出损失训练的模型,其注意力矩阵既具有稀疏性,又能有效捕获相关的输入依赖关系。我们目前正继续开展工作,以证明该方法在更大模型和不同架构上的价值。
英文摘要:
As the context window expands, self-attention increasingly dominates the transformer's inference time. Therefore, accelerating attention computation while minimizing performance degradation is essential for the efficient deployment of Large Language Models (LLMs). In this study we extend a theoretical framework of attention sparsity in LLMs. A customized loss function is designed to enforce the sparsity by restricting the number of top elements in the attention matrix. We perform an initial set of evaluations with GPT-2 to show the effectiveness of our sparsification approach. The attention matrices of the models trained with the proposed loss are both sparse and effective in capturing relevant input dependencies. We now continue working to demonstrate the value of our approach on larger models and different architectures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。