arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.02230cs.LGcs.AI
收藏

广义离散扩散与自校正

Generalized Discrete Diffusion with Self-Correction

National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学)

Linxuan Wang, Ziyi Wang, Yikun Bai, Wei Deng, Guang Lin, Qifan Song

AI总结:

提出自校正离散扩散模型(SCDD),通过显式状态转移和离散时间学习,简化训练噪声调度,消除冗余重掩码步骤,在GPT-2规模上实现高效并行解码并保持生成质量。

摘要:

自校正是保持离散扩散模型中并行采样且性能损失最小的有效技术。先前的工作在推理时或后训练期间探索了自校正;然而,此类方法通常泛化能力有限,并可能损害推理性能。GIDD通过多步BERT风格的均匀吸收目标开创了基于预训练的自校正。然而,GIDD依赖于连续的基于插值的管道,其中均匀转移和吸收掩码之间的交互不透明,这使超参数调整复杂化并阻碍实际性能。在这项工作中,我们提出了一种自校正离散扩散(SCDD)模型,以显式状态转移和直接在离散时间中学习的方式重新表述预训练自校正。我们的框架还简化了训练噪声调度,消除了冗余的重掩码步骤,并完全依赖均匀转移来学习自校正。在GPT-2规模上的实验表明,我们的方法能够实现更高效的并行解码,同时保持生成质量。

英文摘要:

Self-correction is an effective technique for maintaining parallel sampling in discrete diffusion models with minimal performance degradation. Prior work has explored self-correction at inference time or during post-training; however, such approaches often suffer from limited generalization and may impair reasoning performance. GIDD pioneers pretraining-based self-correction via a multi-step BERT-style uniform-absorbing objective. However, GIDD relies on a continuous interpolation-based pipeline with opaque interactions between uniform transitions and absorbing masks, which complicates hyperparameter tuning and hinders practical performance. In this work, we propose a Self-Correcting Discrete Diffusion (SCDD) model to reformulate pretrained self-correction with explicit state transitions and learn directly in discrete time. Our framework also simplifies the training noise schedule, eliminates a redundant remasking step, and relies exclusively on uniform transitions to learn self-correction. Experiments at the GPT-2 scale demonstrate that our method enables more efficient parallel decoding while preserving generation quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑