你的LLM知道未来：揭示其多Token预测潜力
Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential
AI总结：
针对自回归语言模型逐token生成导致的推理速度限制，提出多token预测框架，通过掩码输入、门控LoRA和推测生成等创新，在保持质量的同时实现最高近5倍加速。
摘要：
自回归语言模型受限于其固有的序列特性，每次只能生成一个token。这种范式限制了推理速度和并行性，尤其是在生成的后期阶段，此时文本的方向和语义已相对确定。在这项工作中，我们提出了一个新颖的框架，利用原始自回归语言模型对未来token的固有知识，结合相关技术来实现这一潜力，并实现对多个后续token的同时预测。我们的方法引入了几项关键创新：(1) 掩码输入公式，从共同前缀联合预测多个未来token；(2) 门控LoRA公式，在保持原始LLM功能的同时，使其具备多token预测能力；(3) 轻量级可学习的采样器模块，从预测的未来token中生成连贯序列；(4) 一组辅助训练损失，包括一致性损失，以增强联合生成token的连贯性和准确性；(5) 推测生成策略，在未来以二次方速度扩展token，同时保持高保真度。我们的方法通过对预训练模型进行监督微调实现了显著加速。例如，它生成代码和数学内容的速度提高了近5倍，通用聊天和知识任务的性能提升了近2.5倍。这些提升均未造成任何质量损失。
英文摘要：
Autoregressive language models are constrained by their inherently sequential nature, generating one token at a time. This paradigm limits inference speed and parallelism, especially during later stages of generation when the direction and semantics of text are relatively certain. In this work, we propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens, combining techniques to realize this potential and enable simultaneous prediction of multiple subsequent tokens. Our approach introduces several key innovations: (1) a masked-input formulation where multiple future tokens are jointly predicted from a common prefix; (2) a gated LoRA formulation that preserves the original LLM's functionality, while equipping it for multi-token prediction; (3) a lightweight, learnable sampler module that generates coherent sequences from the predicted future tokens; (4) a set of auxiliary training losses, including a consistency loss, to enhance the coherence and accuracy of jointly generated tokens; and (5) a speculative generation strategy that expands tokens quadratically in the future while maintaining high fidelity. Our method achieves significant speedups through supervised fine-tuning on pretrained models. For example, it generates code and math nearly 5x faster, and improves general chat and knowledge tasks by almost 2.5x. These gains come without any loss in quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
