arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.21768cs.CL
收藏

Zonkey:一种具有可微分分词和概率注意力的分层扩散语言模型

Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention

Alon Rozental(独立研究者)

Alon Rozental

AI总结:

Zonkey是一种通过可微分分词和概率注意力机制实现的分层扩散语言模型,能够从噪声中生成连贯文本并提升领域适应性。

摘要:

大型语言模型(LLMs)已经革新了自然语言处理,但它们仍然受到固定、非可微分分词器(如字节对编码BPE)的限制,这阻碍了端到端优化和对噪声或领域特定数据的适应性。我们介绍了Zonkey,一种分层扩散模型,通过从原始字符到文档级表示的完整可训练流程来解决这些限制。其核心是一个可微分分词器(Segment Splitter),能够学习概率性的开始序列(BOS)决策,使分词能够自适应地生成具有语言意义的边界(例如,单词边界在空格处,句子开始在句号处),而无需显式监督。这种可微性是通过我们新颖的概率注意力机制实现的,该机制结合了位置特定的存在概率,以模拟对理论上无限序列的软掩码,同时保持梯度。序列以概率方式衰减,而不是依赖结束序列标记,支持可变长度输出。分层级别将序列压缩为更高抽象(例如,字符n-gram到词状向量,然后是句状向量),并通过我们的去噪扩散混合模型(DDMM)进行重构,以在潜在空间中实现稳定且高效的去噪。Stitcher确保了在不同段落中的重叠不变性。在维基百科上端到端训练的Zonkey能够从噪声中生成连贯的可变长度文本,展示了涌现的层次结构,并在与基于熵的可学习分词器相比时,显示出对数据分布的有希望的定性对齐。我们的方法向完全基于梯度的LLMs迈进了一步,具有更好的领域适应和可扩展生成的潜力。我们发布了训练和复现实验的源代码。

英文摘要:

Large language models (LLMs) have revolutionized natural language processing, yet they remain constrained by fixed, non-differentiable tokenizers like Byte Pair Encoding (BPE), which hinder end-to-end optimization and adaptability to noisy or domain-specific data. We introduce Zonkey, a hierarchical diffusion model that addresses these limitations through a fully trainable pipeline from raw characters to document-level representations. At its core is a differentiable tokenizer (Segment Splitter) that learns probabilistic beginning-of-sequence (BOS) decisions, enabling adaptive splits that emerge as linguistically meaningful (e.g., word boundaries at spaces, sentence starts at periods) without explicit supervision. This differentiability is enabled by our novel Probabilistic Attention mechanism, which incorporates position-specific existence probabilities to simulate soft masking over theoretically infinite sequences while preserving gradients. Sequences decay probabilistically rather than relying on end-of-sequence tokens, supporting variable-length outputs. Hierarchical levels compress sequences into higher abstractions (e.g., character n-grams to word-like vectors, then sentence-like), with reconstruction via our Denoising Diffusion Mixed Model (DDMM) for stable and efficient denoising in latent space. A Stitcher ensures overlap invariance across segments. Trained end-to-end on Wikipedia, Zonkey generates coherent, variable-length text from noise, demonstrating emergent hierarchies and promising qualitative alignment to data distributions compared to entropy-based learnable tokenizers. Our approach advances toward fully gradient-based LLMs, with potential for better domain adaptation and scalable generation. We release the source code for training and reproducing our experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑