arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00476cs.LGcs.CL
收藏

扩散语言模型可以隐式地近似最优填充长度

Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly

ruc(中国人民大学)

Hengchang Liu, Zhao Yang, Bing Su

AI总结:

CAL方法通过隐式近似最优填充长度,提升代码和文本填充任务的性能,无需额外训练。

摘要:

扩散语言模型(DLMs)提供了一种双向生成框架,自然适合填充任务,但其性能受到预设填充长度的限制。在本文中,我们发现DLMs具有内在能力来发现正确的填充长度。我们识别出在第一步去噪置信度中存在两个关键的统计现象:一个局部的Oracle峰值,出现在真实长度附近,以及一个系统性的长度偏差,常常掩盖这个信号。通过利用这个信号并校准偏差,我们的训练自由方法CAL(校准适应长度)使DLMs能够通过高效搜索在正式解码前近似最优长度。实证评估表明,CAL在代码填充任务中比固定长度基线提高了47.7%,在文本填充任务中比基于聊天的自适应方法提高了40.5%,同时提升了BLEU-2和ROUGE-L分别高达8.5%和9.9%。这些结果表明,CAL为无需任何专门训练的稳健DLM填充开辟了道路。代码可在https://github.com/NiuHechang/Calibrated_Adaptive_Length获得。

英文摘要:

Diffusion language models (DLMs) provide a bidirectional generation framework naturally suited for infilling, yet their performance is constrained by the pre-specified infilling length. In this paper, we reveal that DLMs possess an inherent ability to discover the correct infilling length. We identify two key statistical phenomena in the first-step denoising confidence: a local \textit{Oracle Peak} that emerges near the ground-truth length and a systematic \textit{Length Bias} that often obscures this signal. By leveraging this signal and calibrating the bias, our training-free method \textbf{CAL} (\textbf{C}alibrated \textbf{A}daptive \textbf{L}ength) enables DLMs to approximate the optimal length through an efficient search before formal decoding. Empirical evaluations demonstrate that CAL improves Pass@1 by up to 47.7\% over fixed-length baselines and 40.5\% over chat-based adaptive methods in code infilling, while boosting BLEU-2 and ROUGE-L by up to 8.5\% and 9.9\% in text infilling. These results demonstrate that CAL paves the way for robust DLM infilling without requiring any specialized training. Code is available at https://github.com/NiuHechang/Calibrated_Adaptive_Length.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑