arXiv 2502.099922025-10-18cs.CLcs.LG

大型语言扩散模型

Large Language Diffusion Models

Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）；Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程研究中心）；Ant Group（蚂蚁集团）

Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, Chongxuan Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出从头预训练并经 SFT 的扩散语言模型 LLaDA，通过掩码与反向生成实现可扩展语言建模，性能媲美同规模自回归模型并缓解反转诅咒。

摘要：

大型语言模型（LLM）的能力被广泛认为依赖于自回归模型（ARM）。我们通过提出 LLaDA 来挑战这一观念，它是一个在预训练和监督微调（SFT）范式下从头训练的扩散模型。LLaDA 采用前向数据掩码过程和反向生成过程，由 Transformer 参数化以预测被掩码的 token。它通过优化似然下界，为概率推断提供了一种有原则的生成方法。在通用任务、数学、代码等大量基准测试中，LLaDA 展现出强大的可扩展性，表现与我们自建的 ARM 基线相当。值得注意的是，LLaDA 8B 在上下文学习方面可与 LLaMA3 8B 等强大 LLM 竞争；经过 SFT 后，它在多轮对话等案例研究中表现出令人印象深刻的指令遵循能力。此外，LLaDA 解决了反转诅咒，在反转诗歌补全任务中超过 GPT-4o。我们的发现表明扩散模型在大规模语言建模中的前景，并挑战了上述 LLM 核心能力本质上依赖于 ARM 的普遍假设。项目页面和代码：https://ml-gsai.github.io/LLaDA-demo/。

英文摘要：

The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。