Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Facebook AI Research（脸书人工智能研究院）；Allen Institute for AI（艾伦人工智能研究所）
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.
ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；The University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Sun Yat-Sen University（中山大学）；Tencent（腾讯）；Hong Kong Polytechnic University（香港理工大学）
CLEX: Continuous Length Extrapolation for Large Language Models
Sun Yat-sen University（中山大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Hupan Lab（湖畔实验室）；University of Glasgow（格拉斯哥大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
MALTA – Machine Learning Theory and Applications Lab, PUCRS – Porto Alegre, Brazil（MALTA机器学习理论与应用实验室，PUCRS-波士顿实验室，巴西）；Kunumi Institute, Brazil（昆米研究所，巴西）
