arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.09342cs.CLcs.AI
收藏

面向小型语言模型的潜在多头注意力机制

Latent Multi-Head Attention for Small Language Models

Vizuara AI Labs(Vizuara AI实验室)

Sushant Mehta, Raj Dandekar, Rajat Dandekar, Sreedath Panat

AI总结:

该研究针对小型语言模型探索潜在多头注意力的效率与质量权衡,提出结合旋转位置嵌入的半秩MLA架构,在大幅降低内存、提升推理速度的同时保持甚至超越标准注意力的生成质量。

摘要:

我们首次针对小型语言模型开展潜在多头注意力(MLA)的综合研究,揭示了其在效率与质量间值得关注的权衡关系。我们在10万篇合成故事上训练30M参数的GPT模型,对三种架构变体进行基准测试:标准多头注意力(MHA)、MLA,以及结合旋转位置嵌入的MLA(MLA+RoPE)。核心发现是,采用半秩潜在维度(r = d/2)的MLA+RoPE可将KV缓存内存降低45%,而验证损失仅上升0.3%(质量基本与MHA持平),这对内存受限的部署场景而言是帕累托改进。我们进一步证实RoPE对小模型中的MLA至关重要:没有RoPE时,MLA表现比普通注意力差3-5%;加入RoPE后,其表现比普通注意力优2%。在NVIDIA A100 GPU上的推理基准测试显示,r=d/2的MLA比全秩MLA推理速度提升1.4倍,同时保持内存节省优势。GPT-4评估结果与困惑度结果一致,我们的模型在语法、创造性和一致性指标上获得最高质量评分(7.4/10)。代码和模型将在论文录用后发布。

英文摘要:

We present the first comprehensive study of latent multi-head attention (MLA) for small language models, revealing interesting efficiency-quality trade-offs. Training 30M-parameter GPT models on 100,000 synthetic stories, we benchmark three architectural variants: standard multi-head attention (MHA), MLA, and MLA with rotary positional embeddings (MLA+RoPE). Our key finding is that MLA+RoPE with half-rank latent dimensions (r = d/2) achieves a 45% KV-cache memory reduction while incurring only a 0.3% increase in validation loss (essentially matching MHA quality)- a Pareto improvement for memory constrained deployment. We further show that RoPE is crucial for MLA in small models: without it, MLA underperforms vanilla attention by 3-5%, but with RoPE, it surpasses vanilla by 2%. Inference benchmarks on NVIDIA A100 GPUs reveal that MLA with r=d/2 achieves a 1.4 times speedup over full-rank MLA while maintaining the memory savings. GPT-4 evaluations corroborate perplexity results, with ours achieving the highest quality scores (7.4/10) across grammar, creativity, and consistency metrics. Code and models will be released upon acceptance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑