Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this statement also apply to parameter-efficient MoE? In this paper, we introduce a novel parameter-efficient MoE method, \textit{\textbf{M}oE-L\textbf{o}RA with \textbf{L}ayer-wise Expert \textbf{A}llocation (MoLA)} for Transformer-based models, where each model layer has the flexibility to employ a varying number of LoRA experts. We investigate several architectures with varying layer-wise expert configurations. Experiments on six well-known NLP and commonsense QA benchmarks demonstrate that MoLA achieves equal or superior performance compared to all baselines. We find that allocating more LoRA experts to higher layers further enhances the effectiveness of models with a certain number of experts in total. With much fewer parameters, this allocation strategy outperforms the setting with the same number of experts in every layer. This work can be widely used as a plug-and-play parameter-efficient tuning approach for various applications. The code is available at https://github.com/GCYZSL/MoLA.
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
City University of Hong Kong（香港城市大学）；Tsinghua University（清华大学）；Peking University（北京大学）；University of Michigan（密歇根大学）；University of Chinese Academy of Sciences（中国科学院大学）
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
School of Computer Science and Technology, Beijing Jiaotong University, China（北京交通大学计算机科学与技术学院）；Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China（广西trusted软件重点实验室，桂林电子科技大学）；Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China（北京交通数据挖掘与具身智能重点实验室）；Institute of AI for Industries, Chinese Academy of Sciences, China（工业人工智能研究院，中国科学院）；Nanjing Institute of Software Technology, China（南京软件技术研究院）
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
The University of Hong Kong（香港大学）；Peking University（北京大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of California, Los Angeles（加州大学洛杉矶分校）；National Science Library, Chinese Academy of Sciences（中国科学院国家科学图书馆）
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Science and Technology of China（中国科学技术大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
A Novel Trustworthy Video Summarization Algorithm Through a Mixture of LoRA Experts
School of Information and Network Security, People’s Public Security University of China（中国人民公安大学信息与网络安全学院）；Key Laboratory of Security Prevention Technology and Risk Assessment of the Ministry of Public Security（公安部安全防范技术与风险评估重点实验室）；School of Criminal Investigation, People’s Public Security University of China（中国人民公安大学刑事侦查学院）
Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
Yanshan University（燕山大学）；Delft University of Technology（代尔夫特理工大学）；Shanghai University of Finance and Economics（上海财经大学）；A*STAR Centre for Frontier AI Research（新加坡科技研究局前沿人工智能研究中心）；Nanyang Technological University（南洋理工大学）
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Auditory Cognition and Computational Acoustics Lab（听觉认知与计算声学实验室）；MoE Key Lab of Artificial Intelligence（人工智能MoE重点实验室）；AI Institute School of Computer Science, Shanghai Jiao Tong University（计算机学院人工智能研究所，上海交通大学）；Tencent AI Lab（腾讯AI实验室）
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动的人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
Academy for Science and Design (ASD)（科学设计学院）；Massachusetts Institute of Technology (MIT)（麻省理工学院）
GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute for Advanced Algorithms Research, Shanghai（上海先进算法研究所）；The University of Manchester（曼彻斯特大学）；The University of Pittsburgh（匹兹堡大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Sydney（悉尼大学）
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Southeast University, Nanjing, China（南京东南大学）；Wuhan AI Research, Wuhan, China（武汉人工智能研究院）
P-React: Synthesizing Topic-Adaptive Reactions of Personality Traits via Mixture of Specialized LoRA Experts
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Shanghai Institute of AI for Education, East China Normal University（东华大学人工智能教育研究所）；Xiaohongshu Inc.（小红书公司）
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
School of Computer Science and Software Engineering, Southwest Petroleum University（西南石油大学计算机科学与软件工程学院）；School of Electronic Information and Artificial Intelligence, Leshan Normal University（乐山师范学院电子信息与人工智能学院）
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
