Merging multiple Low-Rank Adaptation (LoRA) modules is promising for constructing general-purpose systems, yet challenging because LoRA update directions span different subspaces and contribute unevenly. When merged naively, such mismatches can weaken the directions most critical to certain task losses while overemphasizing relatively less important ones, ultimately reducing the model's ability to represent all tasks faithfully. We revisit this problem through two perspectives: subspace coverage, which captures how broadly LoRA directions cover diverse representational directions, and anisotropy, which reflects the imbalance of influence across those directions. We propose TARA-Merging (Task-Rank Anisotropy Alignment), which aligns merging weights using a preference-weighted cross-entropy pseudo-loss while preserving task-relevant LoRA subspaces. This ensures broad subspace coverage and mitigates anisotropy via direction-wise reweighting. Across eight vision and six NLI benchmarks, TARA-Merging consistently outperforms vanilla and LoRA-aware baselines, demonstrating strong robustness and generalization, and highlighting the importance of addressing both subspace coverage and anisotropy in LoRA merging.
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
University of Manchester（曼彻斯特大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；InstantX Team（InstantX团队）；University of Edinburgh（爱丁堡大学）；Aalto University（阿alto大学）
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学，上海，中国）；Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China（柔性医疗机器人上海市重点实验室，同仁医院，医疗机器人研究所，上海交通大学，上海，中国）
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
McGill University（麦吉尔大学）；Mila（Mila实验室）；Microsoft（微软公司）；Université de Montréal（蒙特利尔大学）；ServiceNow（ServiceNow公司）；Georgia Institute of Technology（佐治亚理工学院）
Southern University of Science and Technology（南方科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Peng Cheng Laboratory（鹏城实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动的人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
Generalizing the Geometry of Model Merging Through Frechet Averages
Faculty of Computer Science, Dalhousie University（达尔豪斯大学计算机科学学院）；Vector Insitute for Artificial Intelligence（向量人工智能研究所）；Schulich School of Engineering, University of Calgary（卡莱尔大学谢尔奇工程学院）；Department of Computer Science and Software Engineering, Concordia University（康科迪亚大学计算机科学与软件工程系）；Mila - Quebec AI Institute（魁北克人工智能研究所）
Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学）；MoE Key Laboratory of System Control and Information Processing (Shanghai)（系统控制与信息处理MOE重点实验室（上海））
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.（浪潮 Genersoft 有限公司）；Inspur Group Co. Ltd.（浪潮集团有限公司）；Interactive Entertainment Group, Tencent Inc（腾讯公司互动娱乐集团）；Nanjing University of Information Science and Technology（南京信息工程大学）；Northeastern University（东北大学）
C-LoRA: Continual Low-Rank Adaptation for Pre-trained Visual Models
Institute of Intelligent Information Processing, Shanxi University（山西大学智能信息处理研究所）；Alliance Manchester Business School, The University of Manchester（曼彻斯特大学联盟曼彻斯特商学院）
Evolutionary Negative Module Pruning for Better LoRA Merging
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
