循环上下文压缩：高效扩展大语言模型的上下文窗口
Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
AI总结：
本文提出循环上下文压缩方法RCC，在受限存储下高效扩展LLM上下文窗口，并通过指令重建缓解压缩导致的响应质量下降，在32倍压缩率下仍保持接近0.95的BLEU4分数和1M序列长度下近100%的密钥检索准确率。
摘要：
为了扩展基于Transformer的大语言模型（LLMs）的上下文长度并提升理解能力，我们常常面临计算资源和有限内存存储容量的限制。本工作提出了一种名为循环上下文压缩（Recurrent Context Compression，RCC）的方法，旨在在受限存储空间内高效扩展LLMs的上下文窗口长度。我们还研究了下游任务中指令和上下文同时被压缩时模型响应质量变差的问题，并提出一种指令重建方法来缓解该问题。我们在多个任务上验证了所提方法的有效性：在文本重建任务上实现了最高32倍的压缩率，BLEU4分数接近0.95；在序列长度为1M的密钥检索任务上准确率接近100%。最后，与未压缩方法相比，我们的方法在长文本问答任务中展现了具有竞争力的性能，同时在长文本推理任务中显著节省了存储资源。我们的代码、模型和演示可在 https://github.com/WUHU-G/RCC_Transformer 获取。
英文摘要：
To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on text reconstruction tasks with a BLEU4 score close to 0.95, and nearly 100\% accuracy on a passkey retrieval task with a sequence length of 1M. Finally, our method demonstrated competitive performance in long-text question-answering tasks compared to non-compressed methods, while significantly saving storage resources in long-text inference tasks. Our code, models, and demo are available at https://github.com/WUHU-G/RCC_Transformer
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
