CoBit: 基于比特流扩散的语言建模
CoBit: Language Modeling with Bitstream Diffusion
AI总结:
提出CoBit模型,将文本建模为固定宽度二进制比特流上的连续扩散过程,采用匹配滤波残差参数化和基于熵率门控的朗之万校正采样器,在LM1B和OpenWebText上达到接近自回归模型的生成困惑度,并消除词汇表缩放瓶颈。
摘要:
扩散语言模型(DLM)有望实现并行、顺序无关的生成,但在标准基准测试中,它们在样本质量和多样性上历来落后于自回归模型。最近的连续流和扩散方法缩小了这一差距。在这项工作中,我们通过将文本建模为固定宽度二进制比特流上的连续扩散过程,进一步缩小了自回归差距。我们将所得模型称为CoBit(连续比特流扩散)。我们的方法将语义标记表示为模拟比特序列,并使用匹配滤波残差参数化将上下文学习与解析的独立比特后验分离。关键的是,我们采用了一种随机采样器,该采样器应用由熵率分布门控的朗之万型校正,将随机性集中到高信息区域,而在其他区域几乎保持确定性。在LM1B上,我们的130M参数模型在匹配真实数据熵(4.31)的情况下,使用256次神经函数评估(NFE)达到了59.76的生成困惑度(GenPPL),优于先前的DLM基线,并达到了自回归参考水平。在OpenWebText(OWT)上,我们的采样器建立了一个新的连续DLM帕累托前沿,在熵为5.26时实现了27.06的GenPPL,使用的步数比先前1024 NFE基线少4倍。将相同的配方扩展到462M参数模型(CoBit-M)进一步改善了OWT上GenPPL-熵前沿,优于130M模型(CoBit-S)以及中等规模的连续和离散DLM基线,在熵为5.40时达到GenPPL 19.5,接近真实数据熵(5.44),并在高质量区域接近预训练的GPT-2 Medium。作为一个额外的好处,比特流扩散消除了标准DLM的O(V)词汇表缩放瓶颈:通过语义比特修补预测O(log V)逐比特逻辑,它降低了内存并提高了吞吐量,这是一种随着词汇表大小增长而可扩展的范式。
英文摘要:
We introduce CoBit, a continuous diffusion language model that generates text by jointly denoising fixed-width binary token codes. Grouping each code's bits into a single Transformer patch preserves sequence length while replacing vocabulary-wide predictions with $\lceil\log_2 V\rceil$ bit logits. A matched-filter residual parameterization supplies the analytic isolated-bit posterior, leaving the network to learn contextual corrections. Stochastic sampling on an entropy-rate grid provides the largest empirical gain; the grid allocates both solver resolution and effective Langevin correction to noise levels where information is resolved. At 256 denoiser evaluations, CoBit-S (130M parameters) achieves generative perplexity 59.76 on LM1B and 27.06 on OpenWebText, at unigram entropies 4.31 and 5.26, respectively. Scaling to 462M parameters improves OpenWebText generative perplexity to 19.48 at entropy 5.40. Beyond open-ended text generation, CoBit achieves 65.9% exact-match accuracy on hard Sudoku. On GSM8K, score-temperature scaling with an untempered final integration step raises CoBit-S accuracy from 28.1% to 35.3% without additional denoiser evaluations, placing it within the reported range of temperature-tuned discrete diffusion baselines. Matched-trunk profiling also shows lower peak memory and higher throughput than a dense token-output baseline.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。