发表机构
Ecole Polytechnique; Archimedes, Athena RC; University of Crete; IACM-Forth（巴黎综合理工学院; Archimedes，雅典研究与创新中心; 克里特大学; 希腊研究与技术基金会计算医学研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
提出层次化连续扩散语言模型（H-CDLMs），通过并行扩散多粒度令牌表示，以极小开销提升连续扩散语言模型性能，在多个基准上超越基线及同等规模离散模型。
AI 中文摘要
扩散语言模型（DLMs）有望实现与顺序无关的并行文本生成。近年来，通过精心设计的令牌表示和扩散/流空间，连续扩散和流匹配模型取得了显著进展。在这项工作中，我们引入了层次化连续扩散语言模型（H-CDLMs），这是一个简单的框架，以极小的计算和参数开销进一步改进了连续DLMs。借鉴离散DLM和连续图像扩散中关于联合扩散的文献，我们并行扩散多种模态。这些模态代表不同语义粒度的令牌：在我们的实例化中，包括令牌本身以及通过对预训练令牌嵌入进行聚类获得的较粗粒度簇。我们提出了一个通用设置，允许每种模态使用各自的采样器和调度，以增强模态间的相互作用。应用于CoBit，这产生了H-CoBit，在多个基准测试中带来了显著的实证改进。在数据集熵水平上，H-CoBit提高了MAUVE，并在LM1B上达到49.4的生成困惑度（GenPPL），在OWT上达到50.4，相比基线分别提高了24.2和20.7个百分点，甚至超过了同等规模的离散DLMs。在GSM8K上，它达到了27.4%的准确率，优于先前的连续扩散和基于流的模型。我们进一步将H-CDLM应用于流匹配模型FLM，获得了与H-FLM一致的改进，证明了该框架可推广到连续生成范式。我们的代码将在https://this.url上公开提供。
英文摘要
Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities. Applied to CoBit, this yields H-CoBit, which delivers large empirical gains across benchmarks. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 24.2 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior continuous diffusion and flow-based models. We further apply H-CDLM to the flow matching model FLM, obtaining consistent gains with H-FLM and demonstrating that the framework generalizes across continuous generative paradigms. Our code will be made publicly available at https://github.com/matol-16/HCDLM.git .
Comments27 pages, 10 figures
