为何任意顺序自回归模型需要双流注意力：一种结构-语义权衡
Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff
AI总结：
本文提出Decoupled RoPE，通过解耦位置与内容信息，揭示任意顺序生成中结构-语义权衡的本质，表明双流注意力的作用在于解决这一深层权衡而非单纯解耦位置与内容。
摘要：
任意顺序自回归模型（AO-ARMs）通过启用原生键值缓存，为高效的掩码扩散提供了一条有前途的路径，但竞争性的性能迄今为止需要双流注意力，通常被动机为一种将令牌内容与位置解耦的手段。在本文中，我们主张双流注意力可能在更微妙的角色中发挥作用。我们识别出任意顺序生成中的结构-语义权衡：每一步的隐藏表示必须同时关注语义信息丰富的令牌以进行预测，以及结构上最近的令牌以进行总结，这些目标在单一流中竞争注意力容量，但可以在双流中专门化。为了将这种权衡与位置-内容解耦分离，我们提出了Decoupled RoPE，一种对旋转位置嵌入的修改，它在不揭示目标内容的情况下提供目标位置信息。Decoupled RoPE在短序列长度下表现竞争性，其中语义和结构接近性重合，但随着序列长度增加和两种顺序的分歧，其性能下降。这些结果表明，双流注意力的成功不仅仅源于将位置与内容分离，而是源于绕过了任意顺序生成中固有的结构-语义权衡。
英文摘要：
Any-order autoregressive models (AO-ARMs) offer a promising path toward efficient masked diffusion by enabling native key-value caching, but competitive performance has so far required two-stream attention, typically motivated as a means of decoupling token content from position. In this work, we argue that two-stream attention may be serving a more subtle role. We identify a structural-semantic tradeoff in any-order generation: the hidden representation at each step must simultaneously attend to semantically informative tokens for prediction and structurally recent tokens for summarization, objectives that compete for attention capacity in a single stream but can specialize across two streams. To isolate this tradeoff from position-content separation, we propose Decoupled RoPE, a modification to rotary position embeddings that provides target position information without revealing target content. Decoupled RoPE performs competitively at short sequence lengths--where semantic and structural proximity coincide--but degrades as sequence length increases and the two orderings diverge. These results suggest that the success of two-stream attention stems not merely from separating position from content, but from circumventing the deeper structural-semantic tradeoff inherent to any-order generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
