Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging. Although inference-time methods like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) are popular, they face constraints in context budgets, costs, and retrieval fragmentation. Departing from these context-dependent paradigms, this work investigates a parametric approach using Low-Rank Adaptation (LoRA) as a modular knowledge memory. Although few recent works examine this concept, the fundamental mechanics governing its capacity and composability remain largely unexplored. We bridge this gap through the first systematic empirical study mapping the design space of LoRA-based memory, ranging from characterizing storage capacity and optimizing internalization to scaling multi-module systems and evaluating long-context reasoning. Rather than proposing a single architecture, we provide practical guidance on the operational boundaries of LoRA memory. Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.
A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新区(滨江)区块链与数据安全研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);National University of Singapore(新加坡国立大学);Guangdong OPPO Mobile Telecommunications Corp., Ltd.(广东欧珀移动通信有限公司)
Rank-Accuracy Trade-off for LoRA: A Gradient-Flow Analysis
Department of Engineering Sciences and Applied Mathematics, Northwestern University, Evanston, USA(工程科学与应用数学系,西北大学,埃文斯顿,美国);Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, USA(工业工程与管理科学系,西北大学,埃文斯顿,美国)
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学);MoE Key Laboratory of System Control and Information Processing (Shanghai)(系统控制与信息处理MOE重点实验室(上海))
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Petuum Inc.(Petuum公司);Massachusetts Institute of Technology(麻省理工学院)
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
City University of Hong Kong(香港城市大学);Harvard University(哈佛大学);Xi’an Jiaotong University(西安交通大学);Tencent AI Lab(腾讯AI实验室);Zhejiang University(浙江大学);Pengcheng Laboratory(鹏城实验室)
Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China(软件研究所智能软件研究中心,中国科学院,北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京)
Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
University of Science and Technology of China(中国科学技术大学);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
CTR-LoRA: Curvature-Aware and Trust-Region Guided Low-Rank Adaptation for Large Language Models
Cornell University(康奈尔大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Oak Ridge National Laboratory(橡树岭国家实验室);Yale University(耶鲁大学);University of California, Irvine(加州大学欧文分校);Tulane University(路易斯安那大学塔兰帕伊斯分校)