arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.09025cs.CLcs.LG
收藏

Lizard:一种高效的大型语言模型线性化框架

Lizard: An Efficient Linearization Framework for Large Language Models

University of Oregon(俄勒冈大学);Adobe Research(Adobe研究院)
Chien Van Nguyen, Huy Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nik… 展开作者

Chien Van Nguyen, Huy Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen

AI总结:

Lizard提出一种高效线性化框架,将预训练的Transformer大型语言模型转换为亚二次架构,通过亚二次注意力机制提升计算效率,实现模型质量与内存控制的平衡。

摘要:

我们提出了Lizard,一种线性化框架,将预训练的基于Transformer的大型语言模型(LLMs)转换为亚二次架构。Transformer在长序列中面临严重的计算和内存瓶颈,因为softmax注意力的二次复杂性和增长的键值(KV)缓存使推理成为上下文长度受限的内存瓶颈。Lizard通过引入一种亚二次注意力机制,该机制紧密近似softmax注意力同时保持模型质量。与之前受限于固定、非适应性结构的线性化方法不同,Lizard在架构中增加了紧凑、可学习的模块,从而实现适应性的内存控制和稳健的长度泛化。此外,我们引入了一种硬件感知的算法,解决gated attention中的数值不稳定性以加速训练。广泛的实验表明,Lizard实现了接近损失less恢复其教师模型性能,显著优于先前方法,在5-shot MMLU基准上高出9.4-24.5分,并展示了更优的联想回忆能力。

英文摘要:

We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures. Transformers faces severe computational and memory bottlenecks with long sequences due to the quadratic complexity of softmax attention and the growing Key-Value (KV) cache that makes inference memory-bound by context length. Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality. Unlike prior linearization methods constrained by fixed, non-adaptive structures, Lizard augments the architecture with compact, learnable modules that enable adaptive memory control and robust length generalization. Moreover, we introduce a hardwareaware algorithm that solves numerical instability in gated attention to accelerate training. Extensive experiments show that Lizard achieves near-lossless recovery of its teacher model's performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark and demonstrating superior associative recall.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑