arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.05013cs.GRcs.CV
收藏

解开RoPE:多模态和共享注意力中的频率控制

Untwisting RoPE: Frequency Control for Shared Attention in DiTs

Simon Fraser University(西蒙弗雷泽大学);Tel Aviv University(特拉维夫大学);University of Toronto(多伦多大学)

Aryan Mikaeili, Or Patashnik, Andrea Tagliasacchi, Daniel Cohen-Or, Ali Mahdavi-Amiri

AI总结:

本文通过分析RoPE的频率结构,提出方法调节RoPE频率带以实现更符合语义的共享注意力,从而有效控制风格迁移与内容复制的平衡。

摘要:

位置编码对于基于变换器的生成模型至关重要,但它们在多模态和共享注意力设置中的行为尚不完全清楚。在本工作中,我们对旋转位置编码(RoPE)进行了系统分析,证明RoPE自然地分解为具有不同位置敏感性的频率组件。我们展示了这种频率结构解释了为什么共享注意力机制——在生成目标图像的同时关注参考图像的token——会导致参考复制,即模型复制参考内容而不是仅提取其风格线索。我们的分析揭示了RoPE的高频组件主导了注意力计算,迫使查询主要关注空间对齐的参考token,从而引发这种意外的复制行为。基于这些见解,我们提出了一种方法,可以有选择地调节RoPE的频率带,使注意力反映语义相似性而非严格的位置对齐。应用于现代基于变换器的扩散架构中,其中所有token共享注意力,这种调节恢复了稳定且有意义的共享注意力。结果,它使能够有效控制风格转移与内容复制的程度,产生适当的风格对齐生成过程,在其中风格属性被转移而不复制参考内容。

英文摘要:

Positional encodings are essential to transformer-based generative models, yet their behavior in multimodal and attention-sharing settings is not fully understood. In this work, we present a principled analysis of Rotary Positional Embeddings (RoPE), showing that RoPE naturally decomposes into frequency components with distinct positional sensitivities. We demonstrate that this frequency structure explains why shared-attention mechanisms, where a target image is generated while attending to tokens from a reference image, can lead to reference copying, in which the model reproduces content from the reference instead of extracting only its stylistic cues. Our analysis reveals that the high-frequency components of RoPE dominate the attention computation, forcing queries to attend mainly to spatially aligned reference tokens and thereby inducing this unintended copying behavior. Building on these insights, we introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment. Applied to modern transformer-based diffusion architectures, where all tokens share attention, this modulation restores stable and meaningful shared attention. As a result, it enables effective control over the degree of style transfer versus content copying, yielding a proper style-aligned generation process in which stylistic attributes are transferred without duplicating reference content.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑