arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.01097cs.LG
收藏

理解LoRA作为知识记忆:一项实证分析

Understanding LoRA as Knowledge Memory: An Empirical Analysis

New York University(纽约大学)

Seungju Back, Dongwoo Lee, Naun Kang, Taehee Lee, S. K. Hong, Youngjune Gwon, Sungjin Ahn

AI总结:

本文通过系统实证研究,将低秩适配(LoRA)作为模块化知识记忆,探索其存储容量、内部化优化、多模块系统扩展及长上下文推理能力,提供LoRA记忆操作边界的实用指导。

摘要:

预训练大型语言模型(LLM)的持续知识更新日益必要但仍具挑战性。尽管上下文学习(ICL)和检索增强生成(RAG)等推理时方法很流行,但它们面临上下文预算、成本和检索碎片化的限制。脱离这些依赖上下文的范式,本工作研究使用低秩适配(LoRA)作为模块化知识记忆的参数化方法。尽管近期有少量工作探讨了这一概念,但控制其容量和可组合性的基本机制仍很大程度上未被探索。我们通过首个系统性的实证研究来填补这一空白,该研究映射了基于LoRA记忆的设计空间,包括表征存储容量、优化内部化、扩展多模块系统以及评估长上下文推理。我们并非提出单一架构,而是提供关于LoRA记忆操作边界的实用指导。总体而言,我们的发现将LoRA定位为与RAG和ICL互补的记忆轴,具有独特优势。代码和数据集可在 https://github.com/ahn-ml/Understanding-LoRA-as-Knowledge-Memory 获取。

英文摘要:

Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging. Although inference-time methods like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) are popular, they face constraints in context budgets, costs, and retrieval fragmentation. Departing from these context-dependent paradigms, this work investigates a parametric approach using Low-Rank Adaptation (LoRA) as a modular knowledge memory. Although few recent works examine this concept, the fundamental mechanics governing its capacity and composability remain largely unexplored. We bridge this gap through the first systematic empirical study mapping the design space of LoRA-based memory, ranging from characterizing storage capacity and optimizing internalization to scaling multi-module systems and evaluating long-context reasoning. Rather than proposing a single architecture, we provide practical guidance on the operational boundaries of LoRA memory. Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑