arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.02188cs.LG
收藏

多头低秩注意力

Multi-Head Low-Rank Attention

The Pennsylvania State University(宾夕法尼亚州立大学);University of Connecticut(康涅狄格大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Los Angeles(加州大学洛杉矶分校)

Songtao Liu, Hongwu Peng, Zhiwei Zhang, Zhengyu Chen, Yue Guo

AI总结:

本研究提出多头低秩注意力机制,通过高效4路张量并行解码,显著提升大型语言模型的解码速度和性能。

摘要:

大型语言模型中的长上下文推理受到解码阶段键值(KV)缓存加载的瓶颈限制,其中生成过程的序列性要求在每一步都需将KV缓存从离芯片高速内存(HBM)转移到芯片静态随机访问内存(SRAM)。虽然多头潜在注意力(MLA)显著减少了总KV缓存大小,但通过张量并行(TP)进行分布式解码时却受到分片瓶颈的限制。由于其单个潜在头无法被分割,每个设备被迫为每个token重复加载完整的KV缓存,消耗了过多的内存交通,并削弱了TP带来的如权重分片等优势。在本工作中,我们提出了多头低秩注意力(MLRA),它使可分区的潜在状态得以高效地进行4路TP解码。广泛实验表明,MLRA在 perplexity 和下游任务性能方面达到了最先进的水平,同时相较于 MLA 实现了 2.8 倍的解码加速。代码可在 https://github.com/SongtaoLiu0823/MLRA 上获得。预训练权重,以及训练和评估数据,可在 https://huggingface.co/Soughing/MLRA 上获得。

英文摘要:

Long-context inference in large language models is bottlenecked by Key--Value (KV) cache loading during the decoding stage, where the sequential nature of generation requires repeatedly transferring the KV cache from off-chip High-Bandwidth Memory (HBM) to on-chip Static Random-Access Memory (SRAM) at each step. While Multi-Head Latent Attention (MLA) significantly reduces the total KV cache size, it suffers from a sharding bottleneck during distributed decoding via Tensor Parallelism (TP). Since its single latent head cannot be partitioned, each device is forced to redundantly load the complete KV cache for every token, consuming excessive memory traffic and diminishing TP benefits like weight sharding. In this work, we propose Multi-Head Low-Rank Attention (MLRA), which enables partitionable latent states for efficient 4-way TP decoding. Extensive experiments show that MLRA achieves state-of-the-art perplexity and downstream task performance, while also delivering a 2.8$\times$ decoding speedup over MLA. Code is available at https://github.com/SongtaoLiu0823/MLRA. Pretrained weights, along with the training and evaluation data, are available at https://huggingface.co/Soughing/MLRA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑