arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10247cs.CLcs.AI
收藏

QCQA:质量与容量感知的分组查询注意力

QCQA: Quality and Capacity-aware grouped Query Attention

Intel Labs(英特尔实验室)

Vinay Joshi, Prashant Laddha, Shambhavi Sinha, Om Ji Omer, Sreenivas Subramoney

AI总结:

针对 GQA 压缩 KV-cache 却损失生成质量的问题,本文提出 QCQA,用进化算法和低成本适应度函数寻找质量与容量感知的最优查询头分组,在 Llama2 7B 上显著提升准确性并降低缓存需求。

摘要:

键和值特征(KV-cache)过高的内存需求给大语言模型(LLM)的自回归推理带来了重大挑战,限制了文本生成的速度和长度。Multi-Query Attention(MQA,多查询注意力)和 Grouped Query Attention(GQA,分组查询注意力)等方法通过对查询头进行分组,从而减少相应键头和值头的数量,缓解了这些挑战。然而,MQA 和 GQA 以降低 LLM 准确性(文本生成质量)为代价来减小 KV-cache 规模需求。由于缺乏质量感知的查询头分组,这些方法无法确保在 KV-cache 规模与文本生成质量之间实现最优权衡。为解决这一问题,我们提出了 Quality and Capacity-Aware Grouped Query Attention(QCQA,质量与容量感知的分组查询注意力),它利用进化算法以及计算高效、成本低廉的适应度函数来识别最优查询头分组。我们证明,与 GQA 相比,QCQA 在 KV-cache 容量和 LLM 准确性之间实现了显著更优的权衡。对于 Llama2 7B 模型,在未进行微调的情况下,QCQA 在 KV-cache 规模需求相近时,准确性比 GQA 高 20%。在对 QCQA 和 GQA 都进行微调后,在 KV-cache 规模相近的情况下,QCQA 的准确性比 GQA 高 10.55%。此外,在达到相近准确性时,QCQA 所需的 KV-cache 规模比 GQA 少 40%。所提出的质量与容量感知查询头分组可作为自回归 LLM 推理中 KV-cache 优化的新范式。

英文摘要:

Excessive memory requirements of key and value features (KV-cache) present significant challenges in the autoregressive inference of large language models (LLMs), restricting both the speed and length of text generation. Approaches such as Multi-Query Attention (MQA) and Grouped Query Attention (GQA) mitigate these challenges by grouping query heads and consequently reducing the number of corresponding key and value heads. However, MQA and GQA decrease the KV-cache size requirements at the expense of LLM accuracy (quality of text generation). These methods do not ensure an optimal tradeoff between KV-cache size and text generation quality due to the absence of quality-aware grouping of query heads. To address this issue, we propose Quality and Capacity-Aware Grouped Query Attention (QCQA), which identifies optimal query head groupings using an evolutionary algorithm with a computationally efficient and inexpensive fitness function. We demonstrate that QCQA achieves a significantly better tradeoff between KV-cache capacity and LLM accuracy compared to GQA. For the Llama2 $7\,$B model, QCQA achieves $\mathbf{20}$\% higher accuracy than GQA with similar KV-cache size requirements in the absence of fine-tuning. After fine-tuning both QCQA and GQA, for a similar KV-cache size, QCQA provides $\mathbf{10.55}\,$\% higher accuracy than GQA. Furthermore, QCQA requires $40\,$\% less KV-cache size than GQA to attain similar accuracy. The proposed quality and capacity-aware grouping of query heads can serve as a new paradigm for KV-cache optimization in autoregressive LLM inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑