Stable-DiffCoder：推动代码扩散大语言模型的前沿
Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model
AI总结：
Stable-DiffCoder通过基于扩散的训练方法，在代码生成任务中超越了传统自回归模型，提升了代码建模的质量和效率。
摘要：
基于扩散的语言模型（DLLMs）相比自回归（AR）模型提供了非序列、分块生成和更丰富的数据重用，但现有的代码DLLMs在可比预算下仍落后于强大的AR基线。我们在此设置中进行了受控研究，并引入了Stable-DiffCoder，这是一种基于分块扩散的代码模型，重新利用Seed-Coder架构、数据和训练流程。为了实现高效的知识学习和稳定的训练，我们结合了一个增强的分块扩散持续预训练（CPT）阶段，该阶段通过定制的预热和分块裁剪噪声调度进行增强。在相同的数据和架构下，Stable-DiffCoder在广泛的代码基准测试中整体优于其AR counterpart。此外，仅依靠CPT和监督微调阶段，Stable-DiffCoder在广泛的~8B AR和DLLMs上实现了更强的性能，证明了基于扩散的训练可以超越仅AR训练的代码建模质量。此外，基于扩散的任意顺序建模提高了结构化代码的编辑和推理建模，并通过数据增强，受益于低资源编程语言。
英文摘要：
Diffusion-based language models (DLLMs) offer non-sequential, block-wise generation and richer data reuse compared to autoregressive (AR) models, but existing code DLLMs still lag behind strong AR baselines under comparable budgets. We revisit this setting in a controlled study and introduce Stable-DiffCoder, a block diffusion code model that reuses the Seed-Coder architecture, data, and training pipeline. To enable efficient knowledge learning and stable training, we incorporate a block diffusion continual pretraining (CPT) stage enhanced by a tailored warmup and block-wise clipped noise schedule. Under the same data and architecture, Stable-DiffCoder overall outperforms its AR counterpart on a broad suite of code benchmarks. Moreover, relying only on the CPT and supervised fine-tuning stages, Stable-DiffCoder achieves stronger performance than a wide range of \~8B ARs and DLLMs, demonstrating that diffusion-based training can improve code modeling quality beyond AR training alone. Moreover, diffusion-based any-order modeling improves structured code modeling for editing and reasoning, and through data augmentation, benefits low-resource coding languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
