arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.13245cs.CLcs.LG
收藏

GQA:从多头检查点训练广义多查询Transformer模型

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

Google Research(谷歌研究院);University of Southern California(南加利福尼亚大学)

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, Sumit Sanghai

AI总结:

本文提出GQA(分组查询注意力),一种介于多头与多查询注意力之间的键值头配置,并给出利用5%预训练计算将现有检查点升级为GQA的方法,在保持接近多头注意力的质量下获得与MQA相当的速度。

摘要:

多查询注意力(MQA)仅使用单个键值头,大幅加速了解码器推理。然而,MQA可能导致质量下降,而且仅为更快推理而训练单独模型可能并不理想。我们(1)提出了一种配方,利用原始预训练计算的5%,将现有的多头语言模型检查点升级训练为具有MQA的模型;(2)引入了分组查询注意力(GQA),这是多查询注意力的一种泛化,使用中间数量(多于一个、少于查询头数量)的键值头。我们表明,升级训练的GQA在质量上接近多头注意力,同时速度与MQA相当。

英文摘要:

Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑