P-EAGLE：具有可扩展训练的并行草稿EAGLE
P-EAGLE: Parallel-Drafting EAGLE with Scalable Training
AI总结：
P-EAGLE通过可学习共享隐藏状态实现并行多标记预测，结合注意力掩码预计算和序列划分技术，提升长上下文训练效率，实现比自回归EAGLE-3快1.10-1.36倍的速度提升。
摘要：
推理大语言模型生成更长的输出，需要在扩展序列上训练的推测解码草稿者。并行草稿——每个前向传递预测多个标记——相比顺序生成提供了延迟优势，但训练复杂度随着序列长度和并行位置的乘积平方增长，使长上下文训练变得不切实际。我们提出了P（平行）-EAGLE，通过可学习的共享隐藏状态将EAGLE从自回归转换为并行多标记预测。为了扩展到长上下文，我们开发了一个框架，包含注意力掩码预计算和序列划分技术，使可以在单个序列内进行梯度累积的并行预测训练。我们在vLLM上实现了P-EAGLE，并在GPT-OSS 120B、20B和Qwen3-Coder 30B上展示了比自回归EAGLE-3快1.10-1.36倍的速度提升。
英文摘要：
Reasoning LLMs produce longer outputs, requiring speculative decoding drafters trained on extended sequences. Parallel drafting - predicting multiple tokens per forward pass - offers latency benefits over sequential generation, but training complexity scales quadratically with the product of sequence length and parallel positions, rendering long-context training impractical. We present P(arallel)-EAGLE, which transforms EAGLE from autoregressive to parallel multi-token prediction via a learnable shared hidden state. To scale training to long contexts, we develop a framework featuring attention mask pre-computation and sequence partitioning techniques, enabling gradient accumulation within individual sequences for parallel-prediction training. We implement P-EAGLE in vLLM and demonstrate speedups of 1.10-1.36x over autoregressive EAGLE-3 across GPT-OSS 120B, 20B, and Qwen3-Coder 30B.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
