Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational requirements. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey
Technology Innovation Institute(技术创新研究所);Islamic University of Technology(伊斯兰理工大学);Stanford University(斯坦福大学);Amazon GenAI(亚马逊生成式人工智能部门);AI Institute, University of South Carolina(南卡罗来纳大学人工智能研究所)
Beyond the Parameters: A Technical Survey of Contextual Enrichment in Large Language Models: From In-Context Prompting to Causal Retrieval-Augmented Generation
Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
Department of Computing, The Hong Kong Polytechnic University(计算机系,香港理工大学);Ningbo Digital Twin Institute, Eastern Institute of Technology(宁波数字孪生研究院,东部技术研究所)
Beyond Demographics: Enhancing Cultural Value Survey Simulation with Multi-Stage Personality-Driven Cognitive Reasoning
School of Computer Science and Technology, Wuhan University of Science and Technology(武汉科技大学计算机科学与技术学院);Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology(湖北省智能信息处理与实时工业系统重点实验室);Innovation, Policy and Entrepreneurship Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)创新、政策与创业方向);Tübingen AI Center, University of Tübingen(图宾根大学图宾根人工智能中心);Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系)
Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);Wipro AI, Lab45, London, UK(Wipro AI,Lab45,伦敦,英国);School of AI and Data Science, Indian Institute of Technology Jodhpur, India(印度理工学院朱罗普分校人工智能与数据科学学院)
How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond
Sichuan University(四川大学);Singapore Management University(新加坡国立大学);York University(约克大学);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, China(机器学习与产业智能工程研究中心,教育部,中国);National University of Singapore(新加坡国立大学)
Speculative Decoding and Beyond: An In-Depth Survey of Techniques
New York University(纽约大学);University of Pennsylvania(宾夕法尼亚大学);Shenzhen Institute of Information Technology(深圳信息技术研究所);Franklin and Marshall College(弗兰克林与马歇尔学院)
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond
ASUS Intelligent Cloud Services (AICS)(华硕智能云服务(AICS));National University of Singapore(新加坡国立大学);Singapore University of Technology and Design(新加坡科技设计大学)
Beyond Boundaries: A Comprehensive Survey of Transferable Attacks on AI Systems
University of South Florida(佛罗里达州立大学);Missouri University of Science and Technology(密苏里科技大学);Michigan State University(密歇根州立大学);University of Mississippi(密西西比大学);University of Hawaii at Manoa(夏威夷大学曼瑙分校)