Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA;Los Alamos National Laboratory Los Alamos New Mexico USA;Tennessee Tech University Cookeville Tennessee USA;Tennessee Tech University, \ Alamos National Laboratory;Los Alamos National Laboratory;Tennessee Tech University
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
Shanghai Jiao Tong University(上海交通大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);National University of Singapore(新加坡国立大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs
Sichuan University(四川大学);Ant Group(蚂蚁集团);Zhejiang computation(浙江计算);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);CNIC, Chinese Academy of Science(中国科学院计算机网络信息中心)
Rethinking Inter-LoRA Orthogonality in Adapter Merging: Insights from Orthogonal Monte Carlo Dropout
University of Manchester(曼彻斯特大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));InstantX Team(InstantX团队);University of Edinburgh(爱丁堡大学);Aalto University(阿alto大学)
Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学,上海,中国);Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China(柔性医疗机器人上海市重点实验室,同仁医院,医疗机器人研究所,上海交通大学,上海,中国)
ReLoRA: Knowledge-Reusing Adaptation for Fast Rollout of Evolving LLM Services
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州先进研究院,中国科学技术大学)
HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization
The Chinese University of Hong Kong, Hong Kong(香港中文大学);FSI Lab, Huawei Technologies Co., Ltd.(华为技术有限公司FSI实验室);Carnegie Mellon University, Pittsburgh, PA, USA(卡内基梅隆大学)
Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services
Guizhou University of Traditional Chinese Medicine(贵州中医药大学);Guizhou University of Finance and Economics(贵州财经大学);Beijing Technology and Business University(北京工商大学)
LoRA-Pro: Are Low-Rank Adapters Properly Optimized?
University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Nanjing University(南京大学)
GEM-Style Constraints for PEFT with Dual Gradient Projection in LoRA
Department of Computer Science University of North Carolina Greensboro(计算机科学系北卡罗来纳州立大学);Department of Computer Science North Carolina State University(计算机科学系北卡罗来纳州立大学)
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Shanghai AI Laboratory(上海人工智能实验室);Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院)