InfLLM：基于高效上下文记忆的无训练长上下文外推方法
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
AI总结：
提出无训练记忆方法InfLLM，通过外部记忆单元和高效查找机制，使预训练LLM无需微调即可处理超长序列，性能媲美持续训练基线，支持1M长度。
摘要：
大型语言模型（LLMs）已成为处理长流式输入（如基于LLM的智能体）的实际应用中的基石。然而，现有LLM在受限最大长度的序列上预训练，由于域外和干扰问题，无法处理更长的序列。常见解决方案通常涉及在更长序列上进行持续预训练，这会带来昂贵的计算开销和模型能力不可控的变化。本文揭示了LLM在无需微调的情况下理解极长序列的内在能力。为此，我们提出了一种无训练的基于记忆的方法InfLLM。具体而言，InfLLM将远距离上下文存储到额外的记忆单元中，并采用高效机制查找与令牌相关的单元以进行注意力计算。因此，InfLLM允许LLM在有限上下文窗口内高效处理长序列，并很好地捕获长距离依赖。无需任何训练，InfLLM使在数千令牌序列上预训练的LLM能够达到与持续在这些长序列上训练的竞争基线相当的性能。即使序列长度扩展到$1,024$K，InfLLM仍能有效捕获长距离依赖。我们的代码可在\footnote{https://github.com/thunlp/InfLLM}获取。
英文摘要：
Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs (e.g., LLM-driven agents). However, existing LLMs, pre-trained on sequences with a restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training on longer sequences, which will introduce expensive computational overhead and uncontrollable change in model capabilities. In this paper, we unveil the intrinsic capacity of LLMs for understanding extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfLLM. Specifically, InfLLM stores distant contexts into additional memory units and employs an efficient mechanism to lookup token-relevant units for attention computation. Thereby, InfLLM allows LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of a few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences. Even when the sequence length is scaled to $1,024$K, InfLLM still effectively captures long-distance dependencies. Our code can be found in \url{https://github.com/thunlp/InfLLM}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
