Transformer中的加权分组查询注意力
Weighted Grouped Query Attention in Transformers
AI总结:
针对Transformer模型推理成本高的问题,提出加权分组查询注意力WGQA,在T5解码器键值头引入可学习参数实现微调加权平均,性能超GQA且推理无额外开销,还验证了模型缩放定律。
摘要:
注意力机制是Transformer语言模型的基础模块。近期研究表明,扩展模型规模可达到人类水平的性能,但随着模型扩展需求的增长以及硬件内存的限制,这类模型的推理成本仍然居高不下。为缩短推理时间,Multi-Query Attention(MQA,多查询注意力)和Grouped-Query Attention(GQA,分组查询注意力)分别在Shazeer(2019)与Ainslie等人(2023)的研究中被提出。本文提出一种分组查询注意力的变体,名为Weighted Grouped-Query Attention(WGQA,加权分组查询注意力)。我们在T5解码器注意力块的每个键头和值头中引入了新的可学习参数,使模型能够在微调阶段进行加权平均。该模型相比GQA取得了平均0.53%的性能提升,且性能可收敛至传统Multi-head attention(MHA,多头注意力)的水平,同时不会增加推理阶段的额外开销。我们评估发现,这些参数的引入及后续的微调能让模型在训练过程中感知分组机制,从而提升性能。此外,我们通过对比T5-small与T5-base架构的实验结果,在分析中验证了缩放定律。
英文摘要:
The attention mechanism forms the foundational blocks for transformer language models. Recent approaches show that scaling the model achieves human-level performance. However, with increasing demands for scaling and constraints on hardware memory, the inference costs of these models remain high. To reduce the inference time, Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) were proposed in (Shazeer, 2019) and (Ainslieet al., 2023) respectively. In this paper, we propose a variation of Grouped-Query Attention, termed Weighted Grouped-Query Attention (WGQA). We introduced new learnable parameters for each key and value head in the T5 decoder attention blocks, enabling the model to take a weighted average during finetuning. Our model achieves an average of 0.53% improvement over GQA, and the performance converges to traditional Multi-head attention (MHA) with no additional overhead during inference. We evaluated the introduction of these parameters and subsequent finetuning informs the model about the grouping mechanism during training, thereby enhancing performance. Additionally, we demonstrate the scaling laws in our analysis by comparing the results between T5-small and T5-base architecture.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。