具有数千亿参数的大型语言模型（LLMs）已经改变了机器学习领域。然而，在推理时部署这些模型对计算和内存的要求都很高，单个请求可能需要多个GPU和数十GB的内存。多头注意力是LLMs的关键组件之一，可能占LLMs内存和计算需求的50%以上。我们观察到，各个头在关注哪些token上存在大量冗余。基于这一洞察，我们提出了聚类头注意力（Clustered Head Attention, CHAI）。CHAI在运行时将具有高度相关性的头结合起来进行自注意力计算，从而减少内存和计算量。在我们的实验中，我们表明CHAI能够将存储K,V cache的内存需求减少高达21.4%，并将推理时间延迟降低高达1.73倍，且无需任何微调。CHAI在3种不同模型（即OPT-66B、LLAMA-7B、LLAMA-33B）和5个不同评估数据集上实现了最大3.2%的准确率偏差。
英文摘要：
Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is one of the key components of LLMs, which can account for over 50% of LLMs memory and compute requirement. We observe that there is a high amount of redundancy across heads on which tokens they pay attention to. Based on this insight, we propose Clustered Head Attention (CHAI). CHAI combines heads with a high amount of correlation for self-attention at runtime, thus reducing both memory and compute. In our experiments, we show that CHAI is able to reduce the memory requirements for storing K,V cache by up to 21.4% and inference time latency by up to 1.73x without any fine-tuning required. CHAI achieves this with a maximum 3.2% deviation in accuracy across 3 different models (i.e. OPT-66B, LLAMA-7B, LLAMA-33B) and 5 different evaluation datasets.
Iteration Head: A Mechanistic Study of Chain-of-Thought
FAIR, Meta AI（FAIR，Meta人工智能公司）；Datashape, INRIA（Datashape，法国国家信息与自动化研究所）；Courant University and Center for Data Science, NYU（纽约大学柯朗数学科学研究所与数据科学中心）；Center for Data Science, NYU（纽约大学数据科学中心）；NYU（纽约大学）；Courant University（柯朗大学）
Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
Westlake University（西华大学）；McGill University（麦吉尔大学）；Mila - Quebec AI Institute（魁北克AI研究院）；Zhejiang University（浙江大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德智能大学）
Scaling Reasoning Hop Exposes Weaknesses: Demystifying and Improving Hop Generalization in Large Language Models
University of Science and Technology of China（中国科学技术大学）；City University of Hong Kong（香港城市大学）；Zhejiang University（浙江大学）；City University of Hong Kong, Shenzhen Research Institute（香港城市大学深圳研究院）
DEDUCE: Multi-head attention decoupled contrastive learning to discover cancer subtypes based on multi-omics data
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；The Second Xiangya Hospital, Central South University（中南大学湘雅二医院）；National Supercomputing Center in Zhengzhou, Zhengzhou University（郑州大学郑州国家超级计算中心）
提出基于对称多头注意力编码器和解耦对比学习的 DEDUCE 模型，用于多组学癌症数据的无监督特征学习与聚类，实验显示其优于10个深度学习模型及现有方法，并识别出6种 AML 亚型。
