一个潜在变量，多个词元：联合学习压缩嵌入以实现高效语言扩散
One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion
浏览论文内容
中文总结 AI 辅助
针对连续扩散语言模型生成成本高的问题，本文提出JPEG-DLM，通过联合训练压缩器、流匹配模型和解码模块学习结构化压缩嵌入，在LM1B和OWT上取得最低Gen-PPL和最高吞吐量，实现高效生成。
中文摘要 AI 辅助
大多数连续扩散语言模型在每次采样步骤中为每个词元处理一个潜在位置，使得生成成本高昂。两阶段方法通过减少潜在长度来降低成本，但在训练扩散模型之前固定了压缩嵌入空间。来自固定空间的嵌入可能难以用扩散建模并可靠地解码为词元，这限制了压缩后的生成质量。为解决这一问题，我们提出了JPEG-DLM（联合嵌入预测用于扩散语言模型的高效生成），它联合训练一个压缩器、一个流匹配模型和一个解码模块。通过联合嵌入预测，JPEG-DLM学习到的压缩嵌入更具结构性，更容易用扩散建模，并且能可靠地解码为词元。JPEG-DLM在LM1B和OWT数据集上，在最近的扩散和流模型中取得了最低的平均Gen-PPL和最高的吞吐量。在OWT上压缩率为0.5时，其Gen-PPL达到34.52，吞吐量约为ELF的2.3倍。这些结果表明，联合学习压缩嵌入为高效扩散语言建模提供了一条有前景的路径。代码即将发布。
英文摘要
Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be difficult to model with diffusion and decode reliably into tokens, which limits generation quality after compression. To address this problem, we introduce JPEG-DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Language Model), which jointly trains a compressor, a flow matching model and a decoding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens. JPEG-DLM achieves the lowest mean Gen-PPL and highest throughput among recent diffusion and flow models on LM1B and OWT. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3 times ELF's throughput. These results suggest that jointly learning compressed embeddings offers a promising path toward efficient diffusion language modeling. Code will be released soon.
发表机构
- Zhejiang University（浙江大学）
- University of Cambridge（剑桥大学）
- ZJU-UIUC Institute, Zhejiang University（浙江大学伊利诺伊大学厄巴纳-香槟校区联合学院）
机构由 AI 辅助整理，请以论文原文为准。
