BitLM:通过位连续扩散解锁多令牌语言生成
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
AI总结:
BitLM通过位连续扩散模型实现多令牌并行生成,结合因果注意力与联合词决策,提升预训练效率和推理速度,证明单令牌生成并非必要。
摘要:
自回归语言模型逐令牌生成文本,但自然语言本质上是多令牌单元的结构,包括短语、n-gram和词组等共同承载意义的单位。这种单令牌瓶颈限制了预训练期间模型的表达能力和推理时的吞吐量。现有解决方案如推测解码或基于扩散的语言模型要么保留底层瓶颈,要么牺牲语言建模中至关重要的因果结构。我们提出BitLM,一种将每个令牌表示为固定长度二进制码的语言模型,并采用轻量级扩散头在每个块内并行去噪多个令牌。关键在于BitLM在块间保持从左到右的因果注意力,同时在每个块内进行联合词汇决策,结合自回归建模的可靠性与迭代细化的并行性。通过将大词汇量softmax替换为位去噪,BitLM将令牌生成重新框架为紧凑二进制空间中的迭代承诺,从而实现更高效的预训练和显著更快的推理,而无需改变使语言模型有效的因果基础。我们的结果表明,单令牌生成并非基本要求,而是一种接口选择,改变它可得到更强更快的语言模型。我们希望BitLM为下一代语言模型架构指明有前途的方向。
英文摘要:
Autoregressive language models generate text one token at a time, yet natural language is inherently structured in multi-token units, including phrases, n-grams, and collocations that carry meaning jointly. This one-token bottleneck limits both the expressiveness of the model during pre-training and its throughput at inference time. Existing remedies such as speculative decoding or diffusion-based language models either leave the underlying bottleneck intact or sacrifice the causal structure essential to language modeling. We propose BitLM, a language model that represents each token as a fixed-length binary code and employs a lightweight diffusion head to denoise multiple tokens in parallel within each block. Crucially, BitLM preserves left-to-right causal attention across blocks while making joint lexical decisions within each block, combining the reliability of autoregressive modeling with the parallelism of iterative refinement. By replacing the large-vocabulary softmax with bitwise denoising, BitLM reframes token generation as iterative commitment in a compact binary space, enabling more efficient pre-training and substantially faster inference without altering the causal foundation that makes language models effective. Our results demonstrate that the one-token-at-a-time paradigm is not a fundamental requirement but an interface choice, and that changing it can yield a stronger and faster language model. We hope BitLM points toward a promising direction for next-generation language model architectures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。