Rotary Position Embedding (RoPE) is the de facto positional encoding in large language models due to its ability to encode relative positions and support length extrapolation. When adapted to vision transformers, the standard axial formulation decomposes two-dimensional spatial positions into horizontal and vertical components, implicitly restricting positional encoding to axis-aligned directions. We identify this directional constraint as a fundamental limitation of the standard axial 2D RoPE, which hinders the modeling of oblique spatial relationships that naturally exist in natural images. To overcome this limitation, we propose Spiral RoPE, a simple yet effective extension that enables multi-directional positional encoding by partitioning embedding channels into multiple groups associated with uniformly distributed directions. Each group is rotated according to the projection of the patch position onto its corresponding direction, allowing spatial relationships to be encoded beyond the horizontal and vertical axes. Across a wide range of vision tasks including classification, segmentation, and generation, Spiral RoPE consistently improves performance. Qualitative analysis of attention maps further show that Spiral RoPE exhibits more concentrated activations on semantically relevant objects and better respects local object boundaries, highlighting the importance of multi-directional positional encoding in vision transformers.
Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models
Huawei Noah's Ark Lab.(华为诺亚实验室);City University of Hong Kong.(香港城市大学);University of Sydney.(悉尼大学);State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(北京大学人工智能国家重点实验室,智能科学与技术学院)
C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning
State Key Laboratory of Internet of Things for Smart City, University of Macau(物联网智能城市国家重点实验室,澳门大学);Department of Automation, Shanghai Jiaotong University(上海交通大学自动化系);Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院);School of Computer Science and Technology, USTC(中国科学技术大学计算机科学与技术学院);School of Artificial Intelligence and Data Science, USTC(中国科学技术大学人工智能与数据科学学院)
GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models
The University of Sydney(悉尼大学);Edith Cowan University(埃迪斯科文大学);Adelaide University(阿德莱德大学);Wuhan Polytechnic University(武汉轻工大学);Climate, Orange Agricultural Institute(气候研究所,奥兰治农业研究所);La Trobe University(拉筹伯大学)
A Circular Argument : Does RoPE need to be Equivariant for Vision?
Institute of Computer Science and Campus Institute Data Science, University of Göttingen(计算机科学研究所和校园数据科学学院,哥廷根大学);Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany(动态与自组织马克斯·普朗克研究所,哥廷根,德国)
SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs
University of Macau(澳门大学);University of Science and Technology of China(中国科学技术大学);Shanghai Jiaotong University(上海交通大学);Hefei University of Technology(合肥工业大学);National University of Singapore(新加坡国立大学)