DreamOn: 用于代码填充的扩散语言模型超越固定大小画布
DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas
AI总结:
DreamOn通过引入动态长度控制机制,解决了扩散语言模型在代码填充中因固定长度限制导致的性能问题,实现了灵活的可变长度生成。
摘要:
扩散语言模型(DLMs)提供了一种有吸引力的替代方案,相对于自回归模型,提供灵活的、任意顺序的填充,而无需专门的提示设计。然而,它们的实际实用性受到一个关键限制的阻碍:生成需要固定长度的掩码序列。这个限制严重地降低了代码填充性能,当预定义的掩码大小与理想完成长度不匹配时。为了解决这个问题,我们提出了DreamOn,一种新的扩散框架,使动态、可变长度的生成成为可能。DreamOn在扩散过程中加入了两个长度控制状态,使模型能够根据自身的预测自动扩展或收缩输出长度。我们将这一机制整合到现有的DLMs中,仅对训练目标进行最小的修改,而无需改变架构。基于Dream-Coder-7B和DiffuCoder-7B,DreamOn在HumanEval-Infilling和SantaCoder-FIM上实现了与最先进的自回归模型相当的填充性能,并且在真实长度下达到了最佳性能。我们的工作去除了DLMs实际部署的基本障碍,显著提高了它们的灵活性和可变长度生成的应用性。我们的代码可在https://github.com/DreamLM/DreamOn上获得。
英文摘要:
Diffusion Language Models (DLMs) present a compelling alternative to autoregressive models, offering flexible, any-order infilling without specialized prompting design. However, their practical utility is blocked by a critical limitation: the requirement of a fixed-length masked sequence for generation. This constraint severely degrades code infilling performance when the predefined mask size mismatches the ideal completion length. To address this, we propose DreamOn, a novel diffusion framework that enables dynamic, variable-length generation. DreamOn augments the diffusion process with two length control states, allowing the model to autonomously expand or contract the output length based solely on its own predictions. We integrate this mechanism into existing DLMs with minimal modifications to the training objective and no architectural changes. Built upon Dream-Coder-7B and DiffuCoder-7B, DreamOn achieves infilling performance on par with state-of-the-art autoregressive models on HumanEval-Infilling and SantaCoder-FIM and matches oracle performance achieved with ground-truth length. Our work removes a fundamental barrier to the practical deployment of DLMs, significantly advancing their flexibility and applicability for variable-length generation. Our code is available at https://github.com/DreamLM/DreamOn.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。