arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.01766cs.LGcs.AI
收藏

CoMeT:用于高效长上下文建模的协作记忆变换器

CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China(东北大学计算机科学与工程学院);Tsinghua University(清华大学);Future Living Lab of Alibaba(阿里巴巴未来生活实验室)

Runsong Zhao, Shilei Liu, Jiwei Tang, Langming Liu, Haibin Chen, Weidong Zhang, Yujin Yuan, Tong Xiao, Jingbo Zhu, Wenbo Su, Bo Zheng

AI总结:

CoMeT通过双内存系统和层级流水线并行策略,实现长上下文处理的常数内存和线性时间复杂度,有效提升模型性能。

摘要:

标准变换器的二次复杂度和无限增长的键值缓存对长上下文处理构成主要障碍。为克服这一问题,我们引入了协作记忆变换器(CoMeT),一种新的架构,使大语言模型能够以恒定的内存使用和线性时间复杂度处理任意长的序列。CoMeT设计为一个高效、可插拔的模块,可以仅通过最小的微调即可集成到预训练模型中。它在顺序数据块上运行,使用双内存系统管理上下文:一个FIFO队列上的临时内存用于最近的事件,一个带有门控更新规则的全局内存用于长距离依赖。这些内存随后作为下一个块的动态软提示。为了在极长上下文中实现高效的微调,我们引入了一种新的层级流水线并行策略。我们的方法效果显著:一个配备CoMeT并微调于32k上下文的模型可以准确地从1M个token序列中的任何位置检索出密码。在SCROLLS基准测试中,CoMeT超越了其他高效方法,在摘要任务中实现了与全注意基线相当的性能。其实际有效性进一步在现实世界的代理和用户行为问答任务中得到验证。代码可在:https://github.com/LivingFutureLab/Comet

英文摘要:

The quadratic complexity and indefinitely growing key-value (KV) cache of standard Transformers pose a major barrier to long-context processing. To overcome this, we introduce the Collaborative Memory Transformer (CoMeT), a novel architecture that enables LLMs to handle arbitrarily long sequences with constant memory usage and linear time complexity. Designed as an efficient, plug-in module, CoMeT can be integrated into pre-trained models with only minimal fine-tuning. It operates on sequential data chunks, using a dual-memory system to manage context: a temporary memory on a FIFO queue for recent events, and a global memory with a gated update rule for long-range dependencies. These memories then act as a dynamic soft prompt for the next chunk. To enable efficient fine-tuning on extremely long contexts, we introduce a novel layer-level pipeline parallelism strategy. The effectiveness of our approach is remarkable: a model equipped with CoMeT and fine-tuned on 32k contexts can accurately retrieve a passkey from any position within a 1M token sequence. On the SCROLLS benchmark, CoMeT surpasses other efficient methods and achieves performance comparable to a full-attention baseline on summarization tasks. Its practical effectiveness is further validated on real-world agent and user behavior QA tasks. The code is available at: https://github.com/LivingFutureLab/Comet

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑