arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01774cs.LGcs.AI
收藏

FLARE: 混合语言模型的扩散方法

FLARE: Diffusion for Hybrid Language Model

Adobe Research(Adobe研究院);Georgia Institute of Technology(佐治亚理工学院)

Yuchen Zhu, Jing Shi, Chongjian Ge, Hao Tan, Yiran Xu, Wanrong Zhu, Jason Kuen, Koustava Goswami, Rajiv Jain, Yongxin Chen, Molei Tao, Jiuxiang Gu

AI总结:

提出FLARE框架,通过结合自回归和扩散目标、硬件感知内核和统一推理,将混合注意力LLM转换为支持并行解码的扩散模型,在保持能力的同时提升吞吐量。

摘要:

自回归(AR)大型语言模型(LLM)已取得广泛的实际成功,但顺序解码仍然是低延迟部署的关键瓶颈。近期的高效推理工作沿着两个方向推进:通过高效架构降低每次模型调用的成本,以及通过并行生成减少串行解码步骤。混合注意力骨干解决了前者,而扩散语言模型(dLLM)通过迭代并行去噪追求后者。结合这些优势仍然具有挑战性:AR到dLLM的转换通常无法保留种子检查点的能力,并且混合注意力循环状态和掩码约束使得扩散训练和服务变得复杂。我们提出了FLARE,一个针对混合注意力LLM的系统转换框架。我们的分析确定迁移数据质量是能力保留的主要决定因素,其重要性超过损失公式和注意力掩码设计。最终框架结合了token等价的AR和扩散目标、硬件感知内核以及统一推理,使得一个检查点能够同时支持AR风格的验证解码和扩散风格的并行去噪。从强大的AR检查点出发,使用有限的训练后数据,FLARE在模型规模上与领先的开源dLLM竞争,并在单GPU并发服务中相比开源dLLM基线实现了持续的吞吐量提升。我们的结果进一步表明,实际dLLM不仅受限于解码算法,还受限于迁移数据质量和当前块扩散目标的训练低效性,这促使我们联合设计数据、目标、架构和推理系统。

英文摘要:

Autoregressive (AR) large language models (LLMs) have achieved broad practical success, but sequential decoding remains a key bottleneck for low-latency deployment. Recent efficient-inference work has progressed along two axes: reducing the cost of each model invocation through efficient architectures, and reducing serial decoding steps through parallel generation. Hybrid attention backbones address the former, while diffusion language models (dLLMs) pursue the latter via iterative parallel denoising. Combining these advantages remains challenging: AR-to-dLLM conversion often fails to preserve seed-checkpoint capability, and hybrid-attention recurrent states and masking constraints make diffusion training and serving nontrivial. We present FLARE, a systematic conversion framework for hybrid-attention LLMs. Our analysis identifies transfer data quality as the primary determinant of capability preservation, outweighing loss formulation and attention-mask design. The resulting framework combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising. Starting from strong AR checkpoints with limited post-training data, FLARE is competitive with leading open-source dLLMs across model scales and delivers consistent throughput gains over open-source dLLM baselines in single-GPU concurrent serving. Our results further suggest that practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives, motivating joint design of data, objectives, architectures, and inference systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑