arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.00364cs.CL
收藏

面向分布的扩增量化:大型语言模型

Distribution-Aware Companding Quantization of Large Language Models

Athul Radhakrishnan, Siddhant Mohan, Mahima Sachdeva

AI总结:

本研究提出一种多标记预测方法,通过提升样本效率和推理速度,显著提高大型语言模型在编码等生成任务中的性能。

摘要:

大型语言模型,如GPT和Llama,是通过下一个标记预测损失进行训练的。在本工作中,我们提出训练语言模型以一次预测多个未来标记可以提高样本效率。更具体地说,在训练语料库的每个位置,我们要求模型使用n个独立的输出头来预测接下来的n个标记,这些输出头建立在共享的模型主干之上。将多标记预测视为一个辅助训练任务,我们发现对于代码和自然语言模型,无需增加训练时间即可提高下游能力。该方法对更大模型规模越来越有用,并且在多次训练时仍具有吸引力。在生成基准如编码任务中,收益尤为明显,我们的模型在多个百分比点上持续优于强大的基线。我们的13B参数模型在HumanEval上解决12%更多问题,在MBPP上解决17%更多问题,优于相应的下一个标记模型。在小型算法任务上的实验表明,多标记预测有利于诱导头和算法推理能力的发展。此外,使用4个标记预测训练的模型在推理时,即使使用大批次大小,速度也快达3倍。

英文摘要:

Large language models such as GPT and Llama are trained with a next-token prediction loss. In this work, we suggest that training language models to predict multiple future tokens at once results in higher sample efficiency. More specifically, at each position in the training corpus, we ask the model to predict the following n tokens using n independent output heads, operating on top of a shared model trunk. Considering multi-token prediction as an auxiliary training task, we measure improved downstream capabilities with no overhead in training time for both code and natural language models. The method is increasingly useful for larger model sizes and keeps its appeal when training for multiple epochs. Gains are especially pronounced on generative benchmarks like coding, where our models consistently outperform strong baselines by several percentage points. Our 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models. Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. As an additional benefit, models trained with 4-token prediction are up to 3X times faster at inference, even with large batch sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑