arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07524cs.CLcs.AIcs.LG
收藏

简单有效的掩码扩散语言模型

Simple and Effective Masked Diffusion Language Models

Cornell Tech(康奈尔科技学院)

Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, Volodymyr Kuleshov

AI总结:

本文提出一种简单有效的掩码离散扩散方法,通过改进训练方案和推导简化的Rao-Blackwellized目标,显著提升了模型性能,在语言建模基准上逼近自回归方法的困惑度并达到扩散模型的最优水平。

摘要:

虽然扩散模型在生成高质量图像方面表现出色,但先前的工作报告指出,在语言建模中,扩散方法和自回归(AR)方法之间存在显著的性能差距。在这项工作中,我们展示了简单的掩码离散扩散比先前认为的性能更好。我们应用了一种有效的训练方案,改善了掩码扩散模型的性能,并推导出一个简化的 Rao-Blackwellized 目标,带来了额外的性能提升。我们的目标具有简单的形式——它是经典掩码语言建模损失的混合——可用于训练仅编码器语言模型,这些模型支持高效的采样器,包括能够像传统语言模型一样半自回归地生成任意长度文本的采样器。在语言建模基准上,一系列采用现代工程实践训练的掩码扩散模型在扩散模型中实现了新的最优性能,并逼近了 AR 困惑度。我们提供了代码,以及项目页面上的博客文章和视频教程:https://s-sahoo.com/mdlm

英文摘要:

While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, we show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has a simple form -- it is a mixture of classical masked language modeling losses -- and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑