arXiv周末暂无论文更新，休息一下吧，周末愉快~~

arXiv 2610.223302026-10-16cs.AI

为连续扩散语言模型设计Transformer

Designing Transformers for Continuous Diffusion Language Models

Dots Studio, Xiaohongshu（小红书Dots工作室）；Polyadic Research Group（Polyadic研究组）；Peking University（北京大学）；SAGA Lab, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）SAGA实验室）；The University of California, Los Angeles（加利福尼亚大学洛杉矶分校）

Weijian Luo, Yujian Chen, Hao Jiang, Guang Cheng, Tianyang Hu

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对连续扩散语言模型（CDLMs）优化设计了LangDiT架构，在不同CDLM模型上实现了生成困惑度的显著降低，建立了连续语言生成的新state-of-the-art。

摘要：

连续扩散语言模型（CDLMs）为并行语言生成提供了一种替代方法；然而，现有大多数研究直接采用来自因果大语言模型（LLMs）或用于图像生成的扩散Transformer（DiTs）的Transformer架构，这些架构并未针对CDLMs进行优化。本文对适用于CDLMs的Transformer架构开展了系统研究，并提出了语言扩散Transformer（LangDiT），其结合了特征与查询-键归一化、更宽的噪声条件、似然捷径梯度路由与校准，以及置信门控训练自条件。在匹配的LangFlow对比中，LangDiT显著降低了生成困惑度，在不同模型形式的连续语言生成中建立了新的 state-of-the-art。以LangFlow作为代表性CDLM，LangDiT作为即插即用骨干，在OpenWebText数据集上对1024-token序列进行无条件生成时，使用1024个NFEs的生成困惑度（GenPPL）为21.72，使用128个NFEs的GenPPL为44.72。当适配Plaid式CDLM形式时，LangDiT在使用1024个去噪NFEs时实现了11.33的领先GenPPL，使用128个去噪NFEs时实现了27.51的GenPPL。这些改进是在保留各模型底层表示、训练目标和推理采样器的情况下实现的，从而将Transformer架构确定为改进的根本来源。本研究表明架构设计是连续扩散语言模型改进的重要来源。

英文摘要：

Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑