arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.16967cs.CLcs.AI
收藏

MemLong:面向长文本建模的记忆增强检索

MemLong: Memory-Augmented Retrieval for Long Text Modeling

School of Computer Science and Technology, Soochow University(苏州大学计算机科学与技术学院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))

Weijie Liu, Zecheng Tang, Juntao Li, Kehai Chen, Min Zhang

AI总结:

MemLong通过外部检索器与细粒度可控检索注意力机制增强长上下文语言建模,在单个3090 GPU上将上下文长度从4k扩展到80k,并在多个基准上超越现有LLMs。

摘要:

大语言模型(LLMs)的最新进展已在众多领域取得显著成功。然而,由于注意力机制的二次时间与空间复杂度,以及生成过程中键值缓存不断增长的内存消耗,处理长上下文对LLMs而言仍是一项重大挑战。本文提出MemLong:面向长文本生成的记忆增强检索(Memory-Augmented Retrieval for Long Text Generation),该方法通过利用外部检索器进行历史信息检索,以增强长上下文语言建模能力。MemLong将不可微的“ret-mem”模块与部分可训练的解码器-only语言模型相结合,并引入一种细粒度、可控的检索注意力机制,利用语义级相关块。在多个长上下文语言建模基准上的综合评估表明,MemLong始终优于其他最先进的LLMs。更重要的是,MemLong可在单个3090 GPU上将上下文长度从4k扩展到80k。我们的代码可在 https://github.com/Bui1dMySea/MemLong 获取。

英文摘要:

Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs due to the quadratic time and space complexity of attention mechanisms and the growing memory consumption of the key-value cache during generation. This work introduces MemLong: Memory-Augmented Retrieval for Long Text Generation, a method designed to enhance the capabilities of long-context language modeling by utilizing an external retriever for historical information retrieval. MemLong combines a non-differentiable ``ret-mem'' module with a partially trainable decoder-only language model and introduces a fine-grained, controllable retrieval attention mechanism that leverages semantic-level relevant chunks. Comprehensive evaluations on multiple long-context language modeling benchmarks demonstrate that MemLong consistently outperforms other state-of-the-art LLMs. More importantly, MemLong can extend the context length on a single 3090 GPU from 4k up to 80k. Our code is available at https://github.com/Bui1dMySea/MemLong

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑