发表机构
Zhejiang University; Westlake University; Ant Group（浙江大学; 西湖大学; 蚂蚁集团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
发现连续扩散语言模型（如ELF）的低困惑度源于过度重复，提出一维吸引子机制并设计ACE方法消除重复，提升文本质量。
AI 中文摘要
连续扩散语言模型（如ELF）报告了创纪录的低生成困惑度（Gen-PPL）。我们发现一个陷阱：这些模型的重复远多于人类文本，而Gen-PPL奖励而非惩罚这种重复，因此其低分夸大了质量。去除重复后，ELF-B的Gen-PPL从$19.5$升至$27.7$；最小的模型甚至因为重复最多而获得最佳Gen-PPL。我们将重复追溯到其源头：自调节反馈循环中沿\emph{单一方向}的收缩吸引子，该循环将每一步的干净估计输入下一步。由于故障是一维的，一维修复就足够了，我们提出了一种方法。\textbf{ACE}（吸引子-对比-逃逸）从每一步的反馈中减去该单一、无标签方向。在$105$M模型上估计一次，该方向将重复降至接近人类水平，同时保持竞争力，并且几乎不变地转移到$342$M和$652$M模型以及不同采样器；相同的配方在其他架构上也能恢复有用的方向。由于Gen-PPL本身奖励重复，我们转而衡量每种修复生成人类干净文本所需的计算量，其中ACE便宜$1.5$--$5$倍。
英文摘要
Continuous diffusion language models offer an alternative to autoregressive generation, but their generations may suffer from repetition. We find that unconditional generations from ELF, a recent family of continuous diffusion language models, are more repetitive than human text, while Gen-PPL, a common likelihood-based metric, gives lower perplexity to repetitive generations and can conceal this problem while biasing quality evaluation. Our analysis links this behavior to a self-conditioning feedback loop in which clean-embedding predictions are repeatedly carried into subsequent denoising steps, driving representations toward an effectively one-dimensional contractive attractor associated with repetition. Based on this mechanism, we introduce Attractor-Contrast-Escape (ACE), a training-free inference-time intervention that estimates a repetition direction by contrasting denoising paths trapped in repetition with paths relatively free of repetition and subtracts it from the self-conditioning feedback during denoising. Using a direction estimated only once on ELF-B, ACE reduces mean 4-gram self-repetition rate from 7.28% to 4.48%, while retaining competitive results on several text-quality metrics beyond Gen-PPL. The direction remains effective across ELF sizes and inference configurations, and ACE also generalizes to other unconditional self-conditioned continuous diffusion language models. These results identify self-conditioning feedback as a source of repetition in continuous diffusion language models and show that ACE can directly mitigate this repetition during inference.
