CoDAR：连续扩散语言模型比你想象的更强大
CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
AI总结：
CoDAR通过改进的连续扩散模型和上下文自回归解码器，在生成质量上超越了潜在扩散，并与强离散DLMs相竞争。
摘要：
我们研究了为什么尽管连续扩散语言模型（DLMs）具有吸引人的连续生成动态，却在很大程度上落后于离散扩散方法。在受控的令牌恢复研究中，我们发现令牌四舍五入，即从去噪嵌入到令牌的最终投影，是主要的瓶颈。基于这些见解，我们提出了CoDAR（连续扩散与上下文自回归解码器），一种两阶段框架，该框架在嵌入空间中保持扩散的完全连续性，同时学习一个强大的、上下文条件的离散化器：一个自回归Transformer解码器，能够跨关注去噪嵌入序列，并执行上下文化的四舍五入以生成令牌。在LM1B和OpenWebText上的实验表明，CoDAR在生成质量上显著优于潜在扩散，并与强大的离散DLMs相竞争，同时暴露了一个简单的解码器-温度调节器，用于导航流畅性-多样性权衡。
英文摘要：
We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
