LoRA on the Go通过实例级动态选择和合并适配器，在无需额外训练的情况下提升多任务性能，实验证明其在多个NLP基准和数据集上表现优异。
摘要：
低秩适应（LoRA）已成为微调大语言模型的参数高效方法。然而，传统LoRA适配器通常针对单一任务训练，限制了其在实际应用中处理多样且不可预测的输入领域的适用性。在推理阶段，现有方法通过组合多个LoRAs来提升对多样化任务的性能，但通常需要标注数据或额外的任务特定训练，这在大规模情况下成本高昂。在本文中，我们引入LoRA on the Go（LoGo），一种无需训练的框架，能够在实例级别动态选择和合并适配器，而无需任何额外要求。LoGo利用从单次通过LoRA适配器的正向传递中提取的信号，以识别最相关的适配器并确定其贡献。在5个NLP基准、27个数据集和3个模型家族上，LoGo在某些任务上比基于训练的基线高出3.6%的边际，同时在其他任务上保持竞争力，并维持推理吞吐量，突显了其有效性和实用性。
英文摘要：
Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for fine-tuning large language models. However, conventional LoRA adapters are typically trained for a single task, limiting their applicability in real-world settings where inputs may span diverse and unpredictable domains. At inference time, existing approaches combine multiple LoRAs for improving performance on diverse tasks, while usually requiring labeled data or additional task-specific training, which is expensive at scale. In this work, we introduce LoRA on the Go (LoGo), a training-free framework that dynamically selects and merges adapters at the instance level without any additional requirements. LoGo leverages signals extracted from a single forward pass through LoRA adapters, to identify the most relevant adapters and determine their contributions on-the-fly. Across 5 NLP benchmarks, 27 datasets, and 3 model families, LoGo outperforms training-based baselines on some tasks upto a margin of 3.6% while remaining competitive on other tasks and maintaining inference throughput, highlighting its effectiveness and practicality.
Evolutionary Negative Module Pruning for Better LoRA Merging
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
University of Manchester（曼彻斯特大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；InstantX Team（InstantX团队）；University of Edinburgh（爱丁堡大学）；Aalto University（阿alto大学）
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（图像处理与模式识别研究所，自动化与智能感知学院，上海交通大学，上海，中国）；Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China（柔性医疗机器人上海市重点实验室，同仁医院，医疗机器人研究所，上海交通大学，上海，中国）
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
McGill University（麦吉尔大学）；Mila（Mila实验室）；Microsoft（微软公司）；Université de Montréal（蒙特利尔大学）；ServiceNow（ServiceNow公司）；Georgia Institute of Technology（佐治亚理工学院）
Southern University of Science and Technology（南方科技大学）；Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Peng Cheng Laboratory（鹏城实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.（浪潮 Genersoft 有限公司）；Inspur Group Co. Ltd.（浪潮集团有限公司）；Interactive Entertainment Group, Tencent Inc（腾讯公司互动娱乐集团）；Nanjing University of Information Science and Technology（南京信息工程大学）；Northeastern University（东北大学）
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动的人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
Generalizing the Geometry of Model Merging Through Frechet Averages
Faculty of Computer Science, Dalhousie University（达尔豪斯大学计算机科学学院）；Vector Insitute for Artificial Intelligence（向量人工智能研究所）；Schulich School of Engineering, University of Calgary（卡莱尔大学谢尔奇工程学院）；Department of Computer Science and Software Engineering, Concordia University（康科迪亚大学计算机科学与软件工程系）；Mila - Quebec AI Institute（魁北克人工智能研究所）
