arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.02244cs.CLcs.LG
收藏

超越极限:扩展大型语言模型中上下文长度的技术综述

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

Huawei Technologies Canada(华为技术加拿大公司);University of Western Ontario(西安大略大学);Vector Institute for Artificial Intelligence(向量人工智能研究院)

Xindi Wang, Mahsa Salmani, Parsa Omidi, Xiangyu Ren, Mehdi Rezagholizadeh, Armaghan Eshaghi

AI总结:

本综述全面回顾了扩展大型语言模型上下文长度的技术,涵盖架构修改等方法在训练、微调和推理阶段的应用,以提升长序列处理能力并指明未来研究方向。

摘要:

最近,大型语言模型(LLMs)展现出了包括理解上下文、进行逻辑推理和生成响应在内的卓越能力。然而,这是以严格的计算和内存要求为代价实现的,阻碍了它们有效支持长输入序列的能力。本综述对旨在扩展LLMs序列长度从而增强其长上下文理解能力的最新技术与方法进行了全面回顾。具体而言,我们回顾并分类了广泛的技术,包括架构修改,如修改的位置编码和改变的注意力机制,这些技术旨在增强对更长序列的处理,同时避免计算需求的成比例增加。本研究探讨的多种方法可应用于LLMs的不同阶段,即训练、微调和推理。这使得LLMs能够高效处理扩展的序列。最后一节讨论了当前方法的局限性,并对未来的研究方向提出了建议,强调了序列长度在LLMs持续发展中的重要性。

英文摘要:

Recently, large language models (LLMs) have shown remarkable capabilities including understanding context, engaging in logical reasoning, and generating responses. However, this is achieved at the expense of stringent computational and memory requirements, hindering their ability to effectively support long input sequences. This survey provides an inclusive review of the recent techniques and methods devised to extend the sequence length in LLMs, thereby enhancing their capacity for long-context understanding. In particular, we review and categorize a wide range of techniques including architectural modifications, such as modified positional encoding and altered attention mechanisms, which are designed to enhance the processing of longer sequences while avoiding a proportional increase in computational requirements. The diverse methodologies investigated in this study can be leveraged across different phases of LLMs, i.e., training, fine-tuning and inference. This enables LLMs to efficiently process extended sequences. The limitations of the current methodologies is discussed in the last section along with the suggestions for future research directions, underscoring the importance of sequence length in the continued advancement of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑