通过长度正则化改进扩散语言模型的变长生成
Improving Variable-Length Generation in Diffusion Language Models via Length Regularization
AI总结：
本文提出LR-DLLM，通过长度正则化改进扩散语言模型的变长生成能力，提升了在未知长度下的生成效果。
摘要：
扩散大语言模型（DLLMs）本质上不适合变长生成，因为其推理是在固定长度的画布上定义的，并隐含假设了已知的目标长度。当长度未知时，例如在现实中的完成和填充任务中，简单地在遮罩长度之间比较置信度会系统性地产生偏差，导致生成不足或冗余的延续。在本文中，我们表明这种失败源于生成置信度估计中的内在长度诱导偏差，使现有的DLLMs无法可靠地确定生成长度，使变长推理不可靠。为了解决这个问题，我们提出了LR-DLLM，一种针对DLLMs的长度正则化推理框架，将生成长度视为一个显式变量，并在推理时实现可靠的长度确定。它通过显式的长度正则化将语义兼容性与长度诱导的不确定性解耦，从而纠正了偏置的置信度估计。基于此，LR-DLLM能够在不修改底层DLLM或其训练过程的情况下实现生成跨度的动态扩展或收缩。实验显示，LR-DLLM在完全未知长度下实现了51.3%的HumanEvalInfilling Pass@1（比DreamOn高13.4%），并在四个语言的McEval上实现了51.5%的平均Pass@1（比DreamOn高14.3%）.
英文摘要：
Diffusion Large Language Models (DLLMs) are inherently ill-suited for variable-length generation, as their inference is defined on a fixed-length canvas and implicitly assumes a known target length. When the length is unknown, as in realistic completion and infilling, naively comparing confidence across mask lengths becomes systematically biased, leading to under-generation or redundant continuations. In this paper, we show that this failure arises from an intrinsic lengthinduced bias in generation confidence estimates, leaving existing DLLMs without a robust way to determine generation length and making variablelength inference unreliable. To address this issue, we propose LR-DLLM, a length-regularized inference framework for DLLMs that treats generation length as an explicit variable and achieves reliable length determination at inference time. It decouples semantic compatibility from lengthinduced uncertainty through an explicit length regularization that corrects biased confidence estimates. Based on this, LR-DLLM enables dynamic expansion or contraction of the generation span without modifying the underlying DLLM or its training procedure. Experiments show that LRDLLM achieves 51.3% Pass@1 on HumanEvalInfilling under fully unknown lengths (+13.4% vs. DreamOn) and 51.5% average Pass@1 on four-language McEval (+14.3% vs. DreamOn).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
