arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.21035cs.LGcs.CL
收藏

超越自回归:通过时间自蒸馏实现快速大语言模型

Beyond Autoregression: Fast LLMs via Self-Distillation Through Time

Justin Deschenaux, Caglar Gulcehre

AI总结:

本文提出一种针对离散扩散语言模型的时间自蒸馏方法,将推理步数减少32-64倍,使扩散模型能同时生成至少32个token,在1.3B参数规模下生成速度比带KV缓存的自回归模型快至8倍,并在文本质量和LAMBADA基准上超越自回归模型。

摘要:

自回归(AR)大语言模型(LLMs)已在众多任务中展现出显著的成功。然而,AR建模范式存在一定局限;例如,当代自回归大语言模型被训练为一次生成一个token,这可能导致明显的延迟。近期进展表明,搜索和重复采样可以通过在推理过程中利用更多计算资源来提升定理证明、代码生成和对齐等多种应用中的性能。在本研究中,我们证明扩散语言模型能够同时生成至少32个token,同时在文本质量和LAMBADA自然语言理解基准上超越AR模型的表现。这一成果通过一种针对离散扩散模型的新型蒸馏方法实现,该方法将推理步骤数减少了32-64倍。实际上,在13亿参数规模下,扩散模型即使不使用缓存,其token生成速率也可比采用KV缓存的AR模型快至8倍,我们预计加入缓存后将获得进一步改进。此外,我们证明了该方法对参数量高达860M的扩散语言模型同样有效。

英文摘要:

Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive LLMs are trained to generate one token at a time, which can result in noticeable latency. Recent advances have indicated that search and repeated sampling can enhance performance in various applications, such as theorem proving, code generation, and alignment, by utilizing greater computational resources during inference. In this study, we demonstrate that diffusion language models are capable of generating at least 32 tokens simultaneously, while exceeding the performance of AR models in text quality and on the LAMBADA natural language understanding benchmark. This outcome is achieved through a novel distillation method for discrete diffusion models, which reduces the number of inference steps by a factor of 32-64. Practically, at the 1.3B parameters scale, diffusion models, even without caching, can generate tokens at a rate that is up to 8 times faster than AR models employing KV-caching, and we anticipate further improvements with the inclusion of caching. Moreover, we demonstrate the efficacy of our approach for diffusion language models with up to 860M parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑