arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.04096cs.LG
收藏

CORE: 用于扩散语言模型的上下文鲁棒重掩码

CORE: Context-Robust Remasking for Diffusion Language Models

Kevin Zhai, Sabbir Mollah, Zhenyi Wang, Mubarak Shah

AI总结:

CORE通过上下文鲁棒重掩码方法改进扩散模型解码,通过检测上下文敏感性token提升生成稳定性,在推理阶段实现高效修正,优于现有基线方法。

摘要:

标准的掩码扩散模型(MDMs)解码受上下文刚性限制:token基于瞬时高置信度保留,常忽略早期预测缺乏完整上下文。这导致级联效应,初始不一致引导后续生成。现有修正策略依赖静态置信度分数,但这些信号本质上短视;不一致token可能对模型自身显得自信。我们提出上下文鲁棒重掩码(CORE),一种无需训练的推理阶段修正框架。而非信任静态token概率,CORE通过探测token对目标掩码上下文扰动的敏感性识别上下文敏感token。我们将修正正式化为上下文位移上的鲁棒优化目标,并高效近似该目标以优先修正不稳定token。在LLaDA-8B-Base上,CORE在推理和代码基准上实现一致改进,优于计算匹配基线,并将MBPP提升高达9.2个百分点。

英文摘要:

Standard decoding in Masked Diffusion Models (MDMs) is hindered by context rigidity: tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent tokens can appear confident to the model itself. We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision. Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable tokens for revision. On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑