The Rotary Position Embedding (RoPE) is widely used in the attention heads of many large language models (LLM). It rotates dimensions in the query and the key vectors by different angles according to their positions in the input sequence. For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles. We hypothesize that the wide range of rotation angles may prevent LLMs from utilizing those dimensions. To validate this hypothesis, we present a controlled experiment showing that applying RoPE causes low utility of certain dimensions. Our analyses on three LLMs also indicate that these dimensions do not help LLMs do long-context question answering.
The University of Hong Kong(香港大学);University of Michigan, Ann Arbor(密歇根大学安阿伯分校);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所)
Listening to the Wise Few: Query-Key Alignment Unlocks Latent Correct Answers in Large Language Models
Skolkovo Institute of Science and Technology(斯科尔科沃科学技术研究院);AI Foundation and Algorithm Lab(人工智能基金会与算法实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);Artificial Intelligence Research Institute (AIRI)(人工智能研究院);CNRS(法国国家科学研究中心);Université Paris Cité(巴黎西岱大学)
Sliding Window Attention Training for Efficient Large Language Models
City University of Hong Kong(香港城市大学);Xi’an Jiaotong University(西安交通大学);Jarvis Research Center, Tencent YouTu Lab(腾讯优图实验室研究部);Westlake University(西湖大学);University of Science and Technology of China(中国科学技术大学);Southern University of Science and Technology(南方科技大学)
Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(脑启发智能感知与认知教育部重点实验室,中国科学技术大学);Huawei Technologies Co., Ltd.(华为技术有限公司);College of Intelligence and Computing, Tianjin University(智能与计算学院,天津大学)
Efficient Beam Search for Large Language Models Using Trie-Based Decoding
Department of Computer Science, National Chengchi University(国立成功大学计算机科学系);Department of Computer Science, Cornell University(康奈尔大学计算机科学系);Institute of Information Science, Academia Sinica(中华科学研究院资讯研究所)