通过块级局部性训练掩码扩散语言模型
On the Trainability of Masked Diffusion Language Models via Blockwise Locality
AI总结：
研究通过块级局部性改进掩码扩散语言模型，发现其在结构生成任务中稳定性不足，提出Jigsaw和Scatter模型提升性能。
摘要：
基于块级局部性训练的掩码扩散语言模型的可训练性研究显示，标准随机掩码MDMs在结构生成任务中稳定性不足。通过引入Jigsaw和Scatter模型，注入左到右的归纳偏置，强制块内自回归局部性的同时保持块级迭代优化。实验表明，Jigsaw在线性回归任务中与AR-LLM稳定性相当，而在数独任务中表现强劲，而Scatter则保留了扩散模型在路径寻找任务中的规划优势。结果表明，标准随机掩码MDMs即使有块级变体，可能仍是扩散语言模型在有序生成中的次优实现，推动了超越随机掩码的模型发展。
英文摘要：
Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs), yet their optimization can be substantially less stable. We study blockwise MDMs and compare them with AR-LLMs on three controlled tasks that stress different aspects of structured generation: in-context linear regression, graph path-finding, and Sudoku solving. We find that standard random-masking MDMs fail to reliably learn linear regression, exhibit high variance training dynamics on graph path-finding, while outperforming AR-LLMs on Sudoku. To mitigate these instabilities, we propose two locality aware blockwise models, namely Jigsaw and Scatter, that inject left-to-right inductive bias by enforcing autoregressive locality within blocks while preserving iterative refinement at the block level. Empirically, Jigsaw matches AR-LLM stability on linear regression and remains strong on Sudoku, while Scatter retains diffusion's planning advantage on path-finding. Our results indicate that standard random-masking MDMs, even with blockwise variants, may be a suboptimal instantiation of diffusion LMs for ordered generation, motivating models beyond random masking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
