Large Language Models (LLMs) are known to have limited extrapolation ability beyond their pre-trained context window, constraining their application in downstream tasks with lengthy inputs. Recent studies have sought to extend LLMs' context window by modifying rotary position embedding (RoPE), a popular position encoding method adopted by well-known LLMs such as LLaMA, PaLM, and GPT-NeoX. However, prior works like Position Interpolation (PI) and YaRN are resource-intensive and lack comparative experiments to assess their applicability. In this work, we identify the inherent need for LLMs' attention entropy (i.e. the information entropy of attention scores) to maintain stability and introduce a novel extension to RoPE which combines adjusting RoPE's base frequency and scaling the attention logits to help LLMs efficiently adapt to a larger context window. We validate the superiority of our method in both fine-tuning performance and robustness across different context window sizes on various context-demanding tasks. Notably, our method extends the context window of LLaMA-2-7B-Chat to 16,384 with only 100 samples and 6 training steps, showcasing extraordinary efficiency. Finally, we also explore how data compositions and training curricula affect context window extension for specific downstream tasks, suggesting fine-tuning LLMs with lengthy conversations as a good starting point. We release our code and SFT data at https://github.com/GAIR-NLP/Entropy-ABF.
Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);Institute of Intelligent Computing, Alibaba Group(智能计算研究院,阿里巴巴集团)
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Microsoft Corporation(微软公司);School of Computer Science, Peking University(北京大学计算机学院);National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室)
Exploring Context Window of Large Language Models via Decomposed Positional Vectors
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Baichuan Inc.(百川公司);School of Information, Renmin University of China(中国人民大学信息学院);Department of Computer Science, National University of Singapore(新加坡国立大学计算机科学系)
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
National University of Singapore(新加坡国立大学);Nanjing University(南京大学);University of Toronto(多伦多大学);Mila, Québec AI Institute(魁北克人工智能研究所(Mila));Université de Montréal(蒙特利尔大学);Tencent Inc(腾讯公司);Baidu Inc(百度公司);Nanyang Technological University(南洋理工大学)
CLEX: Continuous Length Extrapolation for Large Language Models
Sun Yat-sen University(中山大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Hupan Lab(湖畔实验室);University of Glasgow(格拉斯哥大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,北京通用人工智能研究院)
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search
TJUNLP Lab, College of Intelligence and Computing, Tianjin University(天津大学智能计算学院);School of New Media and Communication, Tianjin University(天津大学新媒体与传播学院)
Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Singapore Management University(新加坡管理大学);Fudan University(复旦大学);National University of Singapore(新加坡国立大学)