LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficiency, thereby solidifying its position as one of the most prevalent fine-tuning techniques. Due to the modular nature of LoRA's plug-and-play plugins, researchers have delved into the amalgamation of multiple LoRAs to empower models to excel across various downstream tasks. Nonetheless, extant approaches for LoRA fusion grapple with inherent challenges. Direct arithmetic merging may result in the loss of the original pre-trained model's generative capabilities or the distinct identity of LoRAs, thereby yielding suboptimal outcomes. On the other hand, Reference tuning-based fusion exhibits limitations concerning the requisite flexibility for the effective combination of multiple LoRAs. In response to these challenges, this paper introduces the Mixture of LoRA Experts (MoLE) approach, which harnesses hierarchical control and unfettered branch selection. The MoLE approach not only achieves superior LoRA fusion performance in comparison to direct arithmetic merging but also retains the crucial flexibility for combining LoRAs effectively. Extensive experimental evaluations conducted in both the Natural Language Processing (NLP) and Vision & Language (V&L) domains substantiate the efficacy of MoLE.
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
City University of Hong Kong（香港城市大学）；Tsinghua University（清华大学）；Peking University（北京大学）；University of Michigan（密歇根大学）；University of Chinese Academy of Sciences（中国科学院大学）
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.（浪潮 Genersoft 有限公司）；Inspur Group Co. Ltd.（浪潮集团有限公司）；Interactive Entertainment Group, Tencent Inc（腾讯公司互动娱乐集团）；Nanjing University of Information Science and Technology（南京信息工程大学）；Northeastern University（东北大学）
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
The University of Hong Kong（香港大学）；Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of California, Los Angeles（加州大学洛杉矶分校）；National Science Library, Chinese Academy of Sciences（中国科学院国家科学图书馆）
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA；Los Alamos National Laboratory Los Alamos New Mexico USA；Tennessee Tech University Cookeville Tennessee USA；Tennessee Tech University, \ Alamos National Laboratory；Los Alamos National Laboratory；Tennessee Tech University
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
School of Computer Science and Technology, Beijing Jiaotong University, China（北京交通大学计算机科学与技术学院）；Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China（广西trusted软件重点实验室，桂林电子科技大学）；Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China（北京交通数据挖掘与具身智能重点实验室）；Institute of AI for Industries, Chinese Academy of Sciences, China（工业人工智能研究院，中国科学院）；Nanjing Institute of Software Technology, China（南京软件技术研究院）
Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction
School of Software Engineering, South China University of Technology, Guangzhou, China（华南理工大学软件学院）；Key Laboratory of Big Data and Intelligent Robot (SCUT), MOE of China（大数据与智能机器人重点实验室）；Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学计算机系）；School of Electrical Engineering, Guangxi University, Nanning, China（广西大学电气工程学院）；Department of Biostatistics & Health Informatics, King’s College London, London, United Kingdom（伦敦国王学院生物统计与健康信息学系）
Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
Yanshan University（燕山大学）；Delft University of Technology（代尔夫特理工大学）；Shanghai University of Finance and Economics（上海财经大学）；A*STAR Centre for Frontier AI Research（新加坡科技研究局前沿人工智能研究中心）；Nanyang Technological University（南洋理工大学）
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Auditory Cognition and Computational Acoustics Lab（听觉认知与计算声学实验室）；MoE Key Lab of Artificial Intelligence（人工智能MoE重点实验室）；AI Institute School of Computer Science, Shanghai Jiao Tong University（计算机学院人工智能研究所，上海交通大学）；Tencent AI Lab（腾讯AI实验室）
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
Department of Data Science & AI, Monash University, Australia（数据科学与人工智能系，墨尔本大学，澳大利亚）；Department of Human Centred Computing, Monash University, Australia（以人为中心计算系，墨尔本大学，澳大利亚）
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；Artificial Intelligence Research Institute, iFLYTEK Company Ltd.（iFLYTEK公司人工智能研究院）
X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
Academy for Science and Design (ASD)（科学设计学院）；Massachusetts Institute of Technology (MIT)（麻省理工学院）
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动的人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
A Novel Trustworthy Video Summarization Algorithm Through a Mixture of LoRA Experts
School of Information and Network Security, People’s Public Security University of China（中国人民公安大学信息与网络安全学院）；Key Laboratory of Security Prevention Technology and Risk Assessment of the Ministry of Public Security（公安部安全防范技术与风险评估重点实验室）；School of Criminal Investigation, People’s Public Security University of China（中国人民公安大学刑事侦查学院）
P-React: Synthesizing Topic-Adaptive Reactions of Personality Traits via Mixture of Specialized LoRA Experts
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Shanghai Institute of AI for Education, East China Normal University（东华大学人工智能教育研究所）；Xiaohongshu Inc.（小红书公司）
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Southeast University, Nanjing, China（南京东南大学）；Wuhan AI Research, Wuhan, China（武汉人工智能研究院）
