Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models (LLMs). However, achieving the right balance of data is crucial to prevent catastrophic forgetting and interference between tasks. To address these limitations and enhance training flexibility, we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and parameter-efficient tuning method designed for multi-task learning with LLMs. In this paper, we start by individually training multiple domain-specific LoRA modules using corresponding supervised corpus data. These LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE). Subsequently, we combine the multiple LoRAs using an explicit routing strategy and introduce domain labels to facilitate multi-task learning, which help prevent interference between tasks and ultimately enhances the performance of each individual task. Furthermore, each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation. Experiments on diverse tasks demonstrate superior and robust performance, which can further promote the wide application of domain-specific LLMs.
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
School of Computer Science and Technology, Beijing Jiaotong University, China（北京交通大学计算机科学与技术学院）；Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China（广西trusted软件重点实验室，桂林电子科技大学）；Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China（北京交通数据挖掘与具身智能重点实验室）；Institute of AI for Industries, Chinese Academy of Sciences, China（工业人工智能研究院，中国科学院）；Nanjing Institute of Software Technology, China（南京软件技术研究院）
A Parameter-Efficient Transfer Learning Approach through Multitask Prompt Distillation and Decomposition for Clinical NLP
Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida（佛罗里达大学医学院健康结果与生物医学信息学系）；Cancer Informatics Shared Resource, University of Florida Health Cancer Center（佛罗里达大学健康癌症中心癌症信息学共享资源）
MoWE-Audio: Multitask AudioLLMs with Mixture of Weak Encoders
Institute for Infocomm Research (I 2 2 {}^{\text{2}} start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT R)（信息与通信研究 institute）；Agency for Science, Technology and Research (A*STAR)（科技研究局）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)（前沿人工智能研究 center）
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
iFLYTEK Co., Ltd（科大讯飞股份有限公司）；University of Pennsylvania（宾夕法尼亚大学）；Lenovo Connect Co., Ltd（联想智联有限公司）；Niuxin Network Technology Co., Ltd（牛信网络科技有限公司）；East China Normal University（华东师范大学）
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.（浪潮 Genersoft 有限公司）；Inspur Group Co. Ltd.（浪潮集团有限公司）；Interactive Entertainment Group, Tencent Inc（腾讯公司互动娱乐集团）；Nanjing University of Information Science and Technology（南京信息工程大学）；Northeastern University（东北大学）
X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
Academy for Science and Design (ASD)（科学设计学院）；Massachusetts Institute of Technology (MIT)（麻省理工学院）
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science（电子电气工程学院，上海工程技术大学）；Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University（计算神经科学与脑启发智能重点实验室，复旦大学）
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University（自动化系，清华大学）；Academy of Medical Engineering and Translational Medicine, Tianjin University（医学工程与转化医学学院，天津大学）
SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
Department of Computer Science, Hong Kong Chu Hai College（香港 Chu Hai 学院计算机科学系）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
Shanghai Jiao Tong University（上海交通大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；National University of Singapore（新加坡国立大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
City University of Hong Kong（香港城市大学）；Tsinghua University（清华大学）；Peking University（北京大学）；University of Michigan（密歇根大学）；University of Chinese Academy of Sciences（中国科学院大学）
CoLA: Cross-Modal Low-rank Adaptation for Multimodal Downstream Tasks
Centre for Vision, Speech and Signal Processing (CVSSP)（视觉、语音和信号处理中心）；University of Surrey（塞维利亚大学）；Surrey Institute for People-Centred AI（以人为本的人工智能研究所）
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Shanghai AI Laboratory（上海人工智能实验室）；Beihang University（北京航空航天大学）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；University of Sydney（悉尼大学）
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Beijing Laboratory of Advanced Information Network（北京先进信息网络实验室）；Beijing Key Laboratory of Network System Architecture and Convergence（北京网络系统架构与融合重点实验室）；Beijing University of Posts and Telecommunications（北京邮电大学）；Department of Electrical and Computer Engineering and Institute for Data Science and Computing（电气与计算机工程系和数据科学与计算研究所）；University of Miami（迈阿密大学）；Information Systems Technology and Design Pillar（信息系统技术与设计支柱）
A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
University of Florida（佛罗里达大学）；University of California, San Francisco（加利福尼亚大学旧金山分校）；Stony Brook University（纽约州立大学石溪分校）；The University of Texas MD Anderson Cancer Center（得克萨斯大学MD安德森癌症中心）
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
