arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2501.07063cs.IRcs.CL
收藏

基于增量学习的检索增强生成(RAG)模型在线更新方法研究

Research on the Online Update Method for Retrieval-Augmented Generation (RAG) Model with Incremental Learning

School of Engineering and Applied Science(工程与应用科学学院);School of Business(商学院);University of Pennsylvania(宾夕法尼亚大学);Stevens Institute of Technology(史蒂文斯理工学院);College of Engineering(工程学院);College of Computer Sciences(计算机科学学院);Peking University(北京大学);Northeastern University(东北大学);Syracuse University(雪城大学);College of Engineering and Computer Science(工程与计算机科学学院)

Yuxin Fan, Yuxiang Wang, Lipeng Liu, Xirui Tang, Na Sun, Zidong Yu

AI总结:

针对语言模型难以及时适配动态新知识的问题,提出基于RAG的在线更新方法,通过动态记忆、分层检索、多阶段生成融合机制实现增量更新,在知识留存和推理精度上优于主流模型。

摘要:

在信息技术飞速发展、数据量呈指数级增长的当下背景下,语言模型在有效应对动态且不断演变的信息格局、实时更新并适配新知识方面面临着重大挑战。本研究基于现有的Retrieval Enhanced Generation(检索增强生成,RAG)模型,提出了一种具备多重创新机制的在线更新方法。首先,利用动态记忆捕获新出现的数据样本,再通过可调优的知识蒸馏策略将其逐步整合到核心模型中。同时,在检索模块引入分层索引与多层门控机制,确保检索内容更具针对性与准确性。最后,在生成阶段针对不同类型的输入构建多阶段网络结构,对各阶段中间表示进行交叉注意力匹配与筛选,保障新旧知识的有效融合与迭代更新。实验结果表明,该方法在知识留存度与推理准确率方面均优于现有主流对比模型。

英文摘要:

In the contemporary context of rapid advancements in information technology and the exponential growth of data volume, language models are confronted with significant challenges in effectively navigating the dynamic and ever-evolving information landscape to update and adapt to novel knowledge in real time. In this work, an online update method is proposed, which is based on the existing Retrieval Enhanced Generation (RAG) model with multiple innovation mechanisms. Firstly, the dynamic memory is used to capture the emerging data samples, and then gradually integrate them into the core model through a tunable knowledge distillation strategy. At the same time, hierarchical indexing and multi-layer gating mechanism are introduced into the retrieval module to ensure that the retrieved content is more targeted and accurate. Finally, a multi-stage network structure is established for different types of inputs in the generation stage, and cross-attention matching and screening are carried out on the intermediate representations of each stage to ensure the effective integration and iterative update of new and old knowledge. Experimental results show that the proposed method is better than the existing mainstream comparison models in terms of knowledge retention and inference accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑