arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.04624cs.CL
收藏

MEMORYLLM:迈向可自我更新的大型语言模型

MEMORYLLM: Towards Self-Updatable Large Language Models

UC, San Diego(加州大学圣迭戈分校);Amazon(亚马逊公司);UC, Los Angeles(加州大学洛杉矶分校)

Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, Jingbo Shang, Julian McAuley

AI总结:

MEMORYLLM通过引入固定大小记忆池实现自我更新,有效整合新知识并长期保留,经近百万次更新无性能退化。

摘要:

现有的大型语言模型(LLMs)在部署后通常保持静态,这可能使得将新知识注入模型变得困难。我们的目标是构建包含大量可自我更新参数的模型,使模型能够有效且高效地整合新知识。为此,我们引入了MEMORYLLM,一个由Transformer和位于Transformer潜在空间内的固定大小记忆池组成的模型。MEMORYLLM能够通过文本知识自我更新,并记住先前注入的知识。我们的评估展示了MEMORYLLM有效整合新知识的能力,这一点通过其在模型编辑基准上的表现得到证实。同时,该模型展现出长期信息保留能力,这通过我们自定义设计的评估和长上下文基准得到验证。即使在近百万次记忆更新后,MEMORYLLM也表现出操作完整性,没有任何性能退化迹象。我们的代码和模型已在https://github.com/wangyu-ustc/MemoryLLM开源。

英文摘要:

Existing Large Language Models (LLMs) usually remain static after deployment, which might make it hard to inject new knowledge into the model. We aim to build models containing a considerable portion of self-updatable parameters, enabling the model to integrate new knowledge effectively and efficiently. To this end, we introduce MEMORYLLM, a model that comprises a transformer and a fixed-size memory pool within the latent space of the transformer. MEMORYLLM can self-update with text knowledge and memorize the knowledge injected earlier. Our evaluations demonstrate the ability of MEMORYLLM to effectively incorporate new knowledge, as evidenced by its performance on model editing benchmarks. Meanwhile, the model exhibits long-term information retention capacity, which is validated through our custom-designed evaluations and long-context benchmarks. MEMORYLLM also shows operational integrity without any sign of performance degradation even after nearly a million memory updates. Our code and model are open-sourced at https://github.com/wangyu-ustc/MemoryLLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑