Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training. Existing length extrapolation methods often suffer from performance degradation or computational inefficiencies. We thereby use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias. This perspective not only subsumes popular approaches such as relative position embeddings and attention-bias moderated approaches but also exposes their inherent limitations in handling long-range dependencies. To address these shortcomings, motivated by our framework, we introduce Adaptive Positional Encoding (APE), which leverages adaptive frequency modulation and an intricately designed decay bias that incorporates linear, logarithmic, and square-root terms. Our theoretical analysis establishes conditions for infinite-context extrapolation, ensuring that the softmax normalization remains well-defined over unbounded sequences while preserving long-distance correlations, entropy boundedness and gradient positional sensitivity. We substantiate our claims with an experimental case study on TinyStories dataset as well as a new synthetic dataset, \emph{Long Tiny Stories} featuring stories up to 32,000 words. Relevant code, dataset and model weights are available at https://anonymous.4open.science/r/Check-2DAD/.
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Alibaba Cloud Computing(阿里云计算);Xiaohongshu Inc.(小红书公司);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs
Digital Research Institute, Enn Group(恩集团数字研究院);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
针对 LLM 超出训练长度后外推能力骤降的问题,本文提出 Mesa-Extrapolation,通过分块三角注意力与 Stair PE 实现低成本、省内存且更快的长度外推。