Excessive memory requirements of key and value features (KV-cache) present significant challenges in the autoregressive inference of large language models (LLMs), restricting both the speed and length of text generation. Approaches such as Multi-Query Attention (MQA) and Grouped Query Attention (GQA) mitigate these challenges by grouping query heads and consequently reducing the number of corresponding key and value heads. However, MQA and GQA decrease the KV-cache size requirements at the expense of LLM accuracy (quality of text generation). These methods do not ensure an optimal tradeoff between KV-cache size and text generation quality due to the absence of quality-aware grouping of query heads. To address this issue, we propose Quality and Capacity-Aware Grouped Query Attention (QCQA), which identifies optimal query head groupings using an evolutionary algorithm with a computationally efficient and inexpensive fitness function. We demonstrate that QCQA achieves a significantly better tradeoff between KV-cache capacity and LLM accuracy compared to GQA. For the Llama2 $7\,$B model, QCQA achieves $\mathbf{20}$\% higher accuracy than GQA with similar KV-cache size requirements in the absence of fine-tuning. After fine-tuning both QCQA and GQA, for a similar KV-cache size, QCQA provides $\mathbf{10.55}\,$\% higher accuracy than GQA. Furthermore, QCQA requires $40\,$\% less KV-cache size than GQA to attain similar accuracy. The proposed quality and capacity-aware grouping of query heads can serve as a new paradigm for KV-cache optimization in autoregressive LLM inference.
Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);AI Lab, The Yangtze River Delta(长江三角洲人工智能实验室);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University College London(伦敦大学学院);The Hong Kong University of Science and Technology(香港科学与技术大学);UCL Centre for Artificial Intelligence(伦敦大学学院人工智能中心)
Efficient Beam Search for Large Language Models Using Trie-Based Decoding
Department of Computer Science, National Chengchi University(国立成功大学计算机科学系);Department of Computer Science, Cornell University(康奈尔大学计算机科学系);Institute of Information Science, Academia Sinica(中华科学研究院资讯研究所)
Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Institute of Trustworthy Embodied Artificial Intelligence, Fudan University(复旦大学可信具身人工智能研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Pengcheng Laboratory(鹏城实验室);Hikvision Inc(海康威视公司)
SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);Xiaomi, Beijing, China(小米(北京,中国));School of Information Management, Wuhan University(武汉大学信息管理学院)
DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
Leiden Institute of Advanced Computer Science (LIACS), Leiden University(莱顿先进计算机科学研究所(LIACS)、莱顿大学);Department of Microelectronics, Delft University of Technology(电子系、代尔夫特理工大学)
KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embedding
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);Xiaomi(小米);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院)