虽然扩散模型在生成高质量图像方面表现出色,但先前的工作报告指出,在语言建模中,扩散方法和自回归(AR)方法之间存在显著的性能差距。在这项工作中,我们展示了简单的掩码离散扩散比先前认为的性能更好。我们应用了一种有效的训练方案,改善了掩码扩散模型的性能,并推导出一个简化的 Rao-Blackwellized 目标,带来了额外的性能提升。我们的目标具有简单的形式——它是经典掩码语言建模损失的混合——可用于训练仅编码器语言模型,这些模型支持高效的采样器,包括能够像传统语言模型一样半自回归地生成任意长度文本的采样器。在语言建模基准上,一系列采用现代工程实践训练的掩码扩散模型在扩散模型中实现了新的最优性能,并逼近了 AR 困惑度。我们提供了代码,以及项目页面上的博客文章和视频教程:https://s-sahoo.com/mdlm
英文摘要:
While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm
Department of Computer Science, Cornell Tech, NYC, USA(康奈尔科技学院计算机科学系);Department of Computer Science, Cornell University, Ithaca, USA(康奈尔大学计算机科学系);School of Computer and Communication Sciences, EPFL Lausanne, Switzerland(洛桑联邦理工学院计算机与通信科学学院);Department of Computer Science, University of Chicago, Illinois(芝加哥大学计算机科学系);NVIDIA, Santa Clara, USA(英伟达)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部);Huawei Noah’s Ark Lab(华为诺亚实验室)
Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
Graduate School of Data Science, Seoul National University(首尔国立大学数据科学研究生院);Department of Biosystems & Biomaterials Science and Engineering, Seoul National University(首尔国立大学生物系统与生物材料科学与工程系)
Masked Diffusion Language Models with Frequency-Informed Training
National Technical University of Athens(国家技术大学雅典);Archimedes RU, Athena RC(阿基米德RU,雅典RC);University of Bern(伯恩大学);Institute of Language and Signal Processing, Athena RC(语言与信号处理研究所,雅典RC)
T$^\star$: Progressive Block Scaling for Masked Diffusion Language Models Through Trajectory Aware Reinforcement Learning
Shanghai Academy of AI for Science(上海人工智能科学研究院);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);School of Mathematical Sciences(数学科学学院);Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学)
Empirical Analysis of Decoding Biases in Masked Diffusion Models
School of Computer Science and Engineering, Northeastern University, China(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Institute for AI, Tsinghua University, China(清华大学人工智能研究院计算机科学与技术系);Alibaba Group, Hangzhou, China(阿里巴巴集团)
Improving Sampling for Masked Diffusion Models via Information Gain
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Massachusetts Institute of Technology(麻省理工学院);Shanghai Jiao Tong University(上海交通大学);Beihang University(北航);College of AI, Tsinghua University(清华大学人工智能学院)