揭示掩码扩散模型中的注意力漂浮机制
Revealing the Attention Floating Mechanism in Masked Diffusion Models
AI总结:
本文揭示了掩码扩散模型中注意力漂浮机制,通过动态分散的注意力锚点和浅层结构感知、深层内容聚焦的机制,解释了其在知识密集型任务中性能优势。
摘要:
掩码扩散模型(MDMs)通过双向注意力和去噪过程,正在缩小与自回归模型(ARMs)之间的性能差距。然而,其内部注意力机制仍处于探索阶段。本文研究了MDMs中的注意力行为,揭示了注意力漂浮现象。与ARMs不同,其中注意力收敛到固定汇点,而MDMs表现出动态分散的注意力锚点,这些锚点在去噪步骤和层之间不断变化。进一步分析揭示了其浅层结构感知、深层内容聚焦的注意力机制:浅层利用漂浮令牌构建全局结构框架,而深层则更侧重于捕捉语义内容。实证表明,这种独特的注意力模式为MDMs强大的上下文学习能力提供了机制解释,使其在知识密集型任务中性能比ARMs高出一倍。所有代码和数据集均可在https://github.com/NEUIR/Attention-Floating获取。
英文摘要:
Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets are available at https://github.com/NEUIR/Attention-Floating.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。