掩码扩散模型中解码偏见的实证分析
Empirical Analysis of Decoding Biases in Masked Diffusion Models
AI总结：
本文通过实证分析揭示了掩码扩散模型中注意力漂浮现象，揭示其浅层结构感知与深层内容聚焦的注意力机制，证明其在知识密集型任务中性能优于自回归模型。
摘要：
掩码扩散模型（MDMs）通过双向注意力和去噪过程，正在缩小与自回归模型（ARMs）之间的性能差距。然而，其内部注意力机制仍处于探索阶段。本文研究了MDMs中的注意力行为，揭示了注意力漂浮现象。不同于ARMs中注意力收敛到固定汇点，MDMs表现出动态分散的注意力锚点，这些锚点在去噪步骤和层之间不断变化。进一步分析揭示其浅层结构感知、深层内容聚焦的注意力机制：浅层利用漂浮标记构建全局结构框架，而深层则更侧重于捕捉语义内容。实证表明，这种独特的注意力模式为MDMs强大的上下文学习能力提供了机制解释，使其在知识密集型任务中性能可达到ARMs的两倍。所有代码均在https://github.com/NEUIR/Uncode上提供。
英文摘要：
Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes are available at https://github.com/NEUIR/Uncode.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
