连续扩散语言模型的设计空间
On the Design Space of Continuous Diffusion Language Models
AI总结：
本研究聚焦连续扩散语言模型（CDLMs）的自条件与嵌入几何两个关键调控因素，提出DoubleCat、SemiCat等新设计，在TinyGSM等任务上实现性能提升，助力推进CDLMs设计空间的研究。
摘要：
连续扩散语言模型（CDLMs）有望将连续流建模的优势引入语言领域，包括加速采样和可控生成。该领域发展迅速，CDLMs的能力已开始与离散扩散模型相媲美。然而，与图像或视频等连续模态不同，我们对训练CDLMs时的关键调控因素的理解仍然有限。本研究聚焦于其中两个调控因素，即自条件和嵌入几何，并阐明了其背后的部分机制，进而提出了更经济有效的设计方案。我们首先证明，自条件通过让模型接触训练期间罕见的类推理状态，纠正了训练/推理不匹配问题。通过一种名为DoubleCat的两级插值器，无需额外前向传播即可恢复这些增益，该插值器将TinyGSM的准确率提升了三倍以上。自条件还有助于识别由大范数高斯分布主导的token，这一问题被称为token可识别性。为直接解决该问题，我们向DoubleCat引入了离散揭示机制，得到了SemiCat，该模型在标准训练超参数和调优采样器下，在TinyGSM上达到了离散扩散级别的性能，准确率达49.94%。在17亿参数规模下，SemiCat在6项基于似然的评估中，有4项优于离散扩散模型，且在无条件生成上超过了独热流模型。在嵌入空间中，我们发现交叉熵会驱动嵌入向解码错误率与独热嵌入相近的几何结构发展。我们还简要探索了一系列其他选项，如混合损坏过程和噪声调度，包括异步噪声调度。我们认为，这一探索可帮助学界更好地理解和推进CDLMs的设计空间。
英文摘要：
Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion. Yet, unlike for continuous modalities such as images or videos, our grasp of the levers that matter when training CDLMs remains limited. In this work, we focus on two of these levers, namely self-conditioning and embedding geometry, and shed light on some of the mechanisms behind them, which in turn suggest cheaper and more effective designs. We begin by showing that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training. These gains can be recovered without an extra forward pass by a two-level interpolant, coined $\mathrm{DoubleCat}$, which improves TinyGSM accuracy more than threefold. Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians$\unicode{x2013}$an issue known as token identifiability. To address it directly, we introduce discrete reveals to $\mathrm{DoubleCat}$, yielding $\mathrm{SemiCat}$, which achieves discrete diffusion-level performance on TinyGSM, reaching $49.94\%$ with standard training hyperparameters and a tuned sampler. At the $1.7$B scale, $\mathrm{SemiCat}$ outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation. On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one. We briefly explore a range of further options, such as hybrid corruption processes and noise schedules, including asynchronous ones. We believe this exploration can help the community better understand and advance the design space of CDLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
