There is growing interest in applying Parameter-Efficient Fine-Tuning (PEFT) to Continual Learning, as it enables faster training, mitigates catastrophic forgetting, and facilitates better adaptation to new tasks. Prior approaches have explored a range of strategies, including extending new LoRA branches for each incoming task, training under orthogonal loss objectives across all task-specific LoRA modules, and employing gating mechanisms to integrate new and existing LoRA modules. However, these methods incur memory costs that grow linearly with the number of tasks, limiting their scalability. More recent approaches investigate training a single LoRA by leveraging asymmetries in its parameterization; however, they often suffer from insufficient representational alignment across the LoRA parameter space. To address these limitations, we propose Stiefel Optimization and Aligned Rotation (SOAR), a two-stage framework for continual LoRA learning. First, we constrain the A matrix to lie on the Stiefel manifold, thereby reducing coordinate misalignment to the space of orthogonal rotation matrices. Second, we introduce an Orthogonal Procrustes alignment procedure for the B matrix to estimate the optimal rotation, coupled with a per-rank retention merging strategy that quantifies directional agreement between representations of new and previously learned tasks. We conduct extensive experiments across multiple models and benchmarks, demonstrating that SOAR achieves strong continual learning performance without incurring additional memory overhead.
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟曼彻斯特商学院)
Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning
College of Computer Science and Artificial Intelligence, Fudan University, China(复旦大学计算机科学与人工智能学院);Singapore Management University, Singapore(新加坡国立管理学院)
Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
East China Normal University(华东师范大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Xiamen University(厦门大学)
SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning
City University of Hong Kong(香港城市大学);Harvard University(哈佛大学);Xi’an Jiaotong University(西安交通大学);Tencent AI Lab(腾讯AI实验室);Zhejiang University(浙江大学);Pengcheng Laboratory(鹏城实验室)
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
Department of XXX, University of YYY, Location, Country(XXX部门,YYY大学,地点,国家);School of ZZZ, Institute of WWW, Location, Country(ZZZ学院,WWW研究所,地点,国家);State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi'an, China(信息服务网络国家重点实验室,电信工程学院,西安电子科技大学,西安,中国);School of Electronic Engineering, Xidian University, Xi'an, China(电子工程学院,西安电子科技大学,西安,中国)
Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning
College of Health Solutions, Arizona State University(亚利桑那州立大学健康解决方案学院);Department of Electrical and Computer Engineering, The University of Texas at Austin(德克萨斯大学奥斯汀分校电气与计算机工程系);School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);The GAME School, Arizona State University(亚利桑那州立大学GAME学院)
Energy-Structured Low-Rank Adaptation for Continual Learning
School of Computer Science and Engineering, Southeast University, Nanjing, China(东南大学计算机科学与工程学院,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用重点实验室(东南大学),教育部,中国);Huawei Technologies, Shenzhen, China(华为技术有限公司,深圳,中国)
LoRA Subtraction for Drift-Resistant Space in Exemplar-Free Continual Learning
School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院);Key Laboratory of Intelligent Assessment Technology for Sustainable Tourism, Ministry of Culture and Tourism, Sun Yat-sen University(文化和旅游部可持续旅游智能评估技术重点实验室);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China(教育部机器智能与先进计算重点实验室)
Few-Shot Continual Learning for 3D Brain MRI with Frozen Foundation Models
Independent Researcher, USA(美国独立研究者);Indiana University at Bloomington, United States(印第安纳大学布卢明顿分校);Independent Researcher, Taiwan(台湾独立研究者);National Taiwan University, Taiwan(台湾国立大学);Boise State University, United States(博伊西州立大学);National Yang Ming Chiao Tung University, Taiwan(台湾阳明交通大学)
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Southeast University, Nanjing, China(南京东南大学);Wuhan AI Research, Wuhan, China(武汉人工智能研究院)
Is Parameter Collision Hindering Continual Learning in LLMs?
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Peng Cheng Laboratory(鹏城实验室);DAMO Academy, Alibaba Group(阿里巴巴达摩院);Hupan Lab(湖畔实验室)
GEM-Style Constraints for PEFT with Dual Gradient Projection in LoRA
Department of Computer Science University of North Carolina Greensboro(计算机科学系北卡罗来纳州立大学);Department of Computer Science North Carolina State University(计算机科学系北卡罗来纳州立大学)
Revisiting Weight Regularization for Low-Rank Continual Learning
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);Computer Vision Center(计算机视觉中心);Universitat Autònoma de Barcelona(巴塞罗那自治大学);School of Instrument Science and Engineering(仪器科学与工程学院);Bernoulli Institute(伯努利学院);University of Groningen(格罗宁根大学);School of Software Engineering(软件工程学院)
Practical Continual Forgetting for Pre-trained Vision Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences(人工智能与机器人中心,香港科学创新研究院,中国科学院);SenseTime Research(商汤科技研究院)
Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
School of Computer Science and Statistics, Trinity College Dublin(都柏林圣三一学院计算机科学与统计学院);Department of Mathematics and Computer Science, Eindhoven University of Technology(埃因霍温理工大学数学与计算机科学系)