Sigma:面向高效语言模型的查询、键和值的差分缩放
Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
AI总结:
该研究针对系统领域提出高效大语言模型Sigma,通过DiffQKV注意力机制差异化优化Q、K、V组件,在6T token上预训练,通用领域性能比肩SOTA,系统领域超GPT-4达52.5%,长上下文推理速度较GQA提升33.36%。
摘要:
我们提出Sigma,这是一款专为系统领域设计的高效大语言模型,其创新架构包含DiffQKV注意力机制,并在我们精心收集的系统领域数据上预训练。DiffQKV注意力通过基于Q、K、V组件对模型性能和效率指标的不同影响,差异化优化注意力机制中的查询(Q)、键(K)和值(V)组件,显著提升Sigma的推理效率。具体而言,我们(1)开展大量实验,证明模型对K和V组件压缩的敏感度存在差异,进而开发出差异化压缩的KV;(2)提出增强型Q,扩展Q头维度,在对推理速度影响极小的情况下提升模型的表示能力。严谨的理论和实证分析表明,DiffQKV注意力大幅提升效率,在长上下文场景中,推理速度比传统分组查询注意力(GQA)最多提升33.36%。我们在6万亿个来自多种来源的token上预训练Sigma,包括195亿个精心收集的系统领域数据,以及1万亿个合成和重写数据。在通用领域,Sigma的性能与其他最先进模型相当;在系统领域,我们引入首个综合基准AIMicius,Sigma在所有任务中表现出色,绝对性能提升最高达52.5%,显著优于GPT-4。
英文摘要:
We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data. DiffQKV attention significantly enhances the inference efficiency of Sigma by optimizing the Query (Q), Key (K), and Value (V) components in the attention mechanism differentially, based on their varying impacts on the model performance and efficiency indicators. Specifically, we (1) conduct extensive experiments that demonstrate the model's varying sensitivity to the compression of K and V components, leading to the development of differentially compressed KV, and (2) propose augmented Q to expand the Q head dimension, which enhances the model's representation capacity with minimal impacts on the inference speed. Rigorous theoretical and empirical analyses reveal that DiffQKV attention significantly enhances efficiency, achieving up to a 33.36% improvement in inference speed over the conventional grouped-query attention (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various sources, including 19.5B system domain data that we carefully collect and 1T tokens of synthesized and rewritten data. In general domains, Sigma achieves comparable performance to other state-of-arts models. In the system domain, we introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates remarkable performance across all tasks, significantly outperforming GPT-4 with an absolute improvement up to 52.5%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。