arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.11819cs.CL
收藏

面向大型语言模型的按头可共享注意力

Head-wise Shareable Attention for Large Language Models

Shanghai Jiao Tong University(上海交通大学);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海市Web3可信数据流通与治理重点实验室)

Zouying Cao, Yifei Yang, Hai Zhao

AI总结:

针对 LLM 部署中的内存压力,本文提出按注意力头共享参数的 DirectShare 与 PostShare 两种方法,实验表明细粒度权重共享可在保持能力的同时降低内存占用。

摘要:

大型语言模型(LLM)参数量巨大,这限制了它们在边缘设备上的部署。权重共享是一种很有前景的解决方案,它鼓励权重复用,能够以较小的性能下降有效降低内存占用。然而,当前的权重共享技术主要关注 BERT 等小规模模型,并采用逐层共享等粗粒度共享规则。鉴于 LLM 的普及,这一点变得具有局限性,而且共享整个层或块显然会削弱权重共享的灵活性。本文提出了一种面向大型语言模型的按头可共享注意力视角。我们进一步提出了两种在注意力头之间共享参数的内存高效方法,并特别针对 LLM。这两种方法都使用相同的动态策略来选择共享权重矩阵。第一种方法无需重新训练即可直接复用预训练权重,记为 DirectShare;第二种方法先在权重矩阵相似性约束下进行后训练,然后再共享,记为 PostShare。实验结果表明,我们的按头共享模型仍保持令人满意的能力,证明了将细粒度权重共享应用于 LLM 的可行性。

英文摘要:

Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. Weight sharing is one promising solution that encourages weight reuse, effectively reducing memory usage with less performance drop. However, current weight sharing techniques primarily focus on small-scale models like BERT and employ coarse-grained sharing rules, e.g., layer-wise. This becomes limiting given the prevalence of LLMs and sharing an entire layer or block obviously diminishes the flexibility of weight sharing. In this paper, we present a perspective on head-wise shareable attention for large language models. We further propose two memory-efficient methods that share parameters across attention heads, with a specific focus on LLMs. Both of them use the same dynamic strategy to select the shared weight matrices. The first method directly reuses the pre-trained weights without retraining, denoted as $\textbf{DirectShare}$. The second method first post-trains with constraint on weight matrix similarity and then shares, denoted as $\textbf{PostShare}$. Experimental results reveal our head-wise shared models still maintain satisfactory capabilities, demonstrating the feasibility of fine-grained weight sharing applied to LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑