发表机构
National University of Science and Technology POLITEHNICA Bucharest（布加勒斯特理工大学国家科技大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究扩散语言模型（DLM）中归纳机制，对比仅注意力AR模型和吸收掩码DLM。发现DLM学习双向归纳电路，方向对称，掩码两侧可见时归纳更强，还证明其能计算掩码令牌比例作隐式时间步长。
AI 中文摘要
虽然自回归（AR）变压器的内部机制已被广泛研究，但对于通过迭代去噪生成文本的新兴替代方案扩散语言模型（DLM）却知之甚少。在这项工作中，我们研究了DLM如何实现归纳，这是上下文学习背后的一种机制，模型在其中找到重复的上下文并复制其后跟随的令牌。我们的分析比较了具有匹配架构的仅注意力AR模型和吸收掩码DLM。我们发现DLM学习了一个双向归纳电路，其中前一个令牌和下一个令牌头部将局部上下文写入残差流，随后的归纳头部使用它从匹配的源位置找到并复制答案。该电路是方向对称的，无论源出现在过去还是未来都能工作。当仅可见左上下文时，与AR模型所见匹配，DLM在归纳能力上并不优于其AR对应模型。然而，我们观察到当掩码令牌的两侧都可见时，它具有更强的归纳能力，这指向双向上下文访问而不是更强的单侧机制。除了归纳，我们还提供了因果证据，表明DLM计算掩码令牌的全局比例并将其用作隐式时间步长，即使它们没有被给予明确的时间步长嵌入。
英文摘要
While the internal mechanisms of autoregressive (AR) transformers have been studied extensively, much less is known about diffusion language models (DLMs), an emerging alternative that generates text by iterative denoising. In this work, we study how DLMs implement induction, a mechanism behind in-context learning in which the model finds a repeated context and copies the token that followed it. Our analysis compares attention-only AR models and absorbing-mask DLMs with matched architectures. We find that DLMs learn a bidirectional induction circuit, where previous-token and next-token heads write local context into the residual stream and later induction heads use it to find and copy the answer from the matching source position. The circuit is direction-symmetric, working whether the source appears in the past or in the future. When only left context is visible, matching what an AR model sees, the DLM does not outperform its AR counterpart in induction capabilities. However, we observe it has stronger induction when both sides of the masked token are visible, pointing to bidirectional context access rather than a stronger one-sided mechanism. Beyond induction, we provide causal evidence that DLMs compute the global fraction of masked tokens and use it as an implicit timestep, even though they are given no explicit timestep embedding.
