Scaling Beyond Masked Diffusion Language Models
超越掩码扩散语言模型的扩展
机构 * Department of Computer Science, Cornell Tech, NYC, USA(康奈尔科技学院计算机科学系) ; Department of Computer Science, Cornell University, Ithaca, USA(康奈尔大学计算机科学系) ; School of Computer and Communication Sciences, EPFL Lausanne, Switzerland(洛桑联邦理工学院计算机与通信科学学院) ; Department of Computer Science, University of Chicago, Illinois(芝加哥大学计算机科学系) ; NVIDIA, Santa Clara, USA(英伟达)
AI总结 本文研究了统一状态和插值离散扩散方法的扩展定律,发现掩码扩散在FLOPs效率上可提升12%,并展示了统一状态扩散在GSM8K任务中的优越表现
Comments code: https://github.com/s-sahoo/scaling-dllms