Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory usage and training cost. This undermines the core goal of parameter-efficient fine-tuning. We propose Monkey Jump, a method that brings mixture-of-experts-style specialization to parameter-efficient fine-tuning without introducing extra trainable parameters for experts or routers. Instead of adding new adapters as experts, Monkey Jump treats the adapters already present in each Transformer block (such as query, key, value, up, and down projections) as implicit experts and routes tokens among them. Routing is performed using k-means clustering with exponentially moving averaged cluster centers, requiring no gradients and no learned parameters. We theoretically show that token-wise routing increases expressivity and can outperform shared adapters by avoiding cancellation effects. Across multi-task experiments covering 14 text, 14 image, and 19 video benchmarks, Monkey Jump achieves competitive performance with mixture-of-experts-based parameter-efficient fine-tuning methods while using 7 to 29 times fewer trainable parameters, up to 48 percent lower memory consumption, and 1.5 to 2 times faster training. Monkey Jump is architecture-agnostic and can be applied to any adapter-based parameter-efficient fine-tuning method.
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
Tsinghua University(清华大学);Beijing Information Science and Technology University(北京信息科技大学);National University of Singapore(新加坡国立大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Beijing University of Posts and Telecommunications(北京邮电大学)
LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
Zhejiang University of Technology(浙江工业大学);The City University of New York(纽约城市大学);Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology(浙江工业大学滨江人工智能研究院);A*STAR(新加坡科技研究局)
OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(复杂系统认知与决策智能重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
The University of Hong Kong(香港大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of California, Los Angeles(加州大学洛杉矶分校);National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆)
HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Models
Department of Computer Science and Technlogy(计算机科学与技术系);Tsinghua University(清华大学);School of Computer Science and Technlogy(计算机科学与技术系);Xi’an Jiaotong University(西安交通大学);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高科技区(滨江)区块链与数据安全研究院)
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Division of Statistics and Machine Learning (STIMA), Department of Computer and Information Science (IDA), Linköping University(统计与机器学习系(STIMA)、计算机与信息科学系(IDA)、利厄普堡大学)
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA;Los Alamos National Laboratory Los Alamos New Mexico USA;Tennessee Tech University Cookeville Tennessee USA;Tennessee Tech University, \ Alamos National Laboratory;Los Alamos National Laboratory;Tennessee Tech University
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.(浪潮 Genersoft 有限公司);Inspur Group Co. Ltd.(浪潮集团有限公司);Interactive Entertainment Group, Tencent Inc(腾讯公司互动娱乐集团);Nanjing University of Information Science and Technology(南京信息工程大学);Northeastern University(东北大学)
LPT++: Efficient Training on Mixture of Long-tailed Experts
School of Computer Science and Technology, Harbin Institute of Technology(哈尔滨工业大学计算机科学与技术学院);School of Computing and Information Systems, Singapore Management University(新加坡管理大学计算与信息系统学院);The Hong Kong Polytechnic University(香港理工大学)
Advancing Enterprise Spatio-Temporal Forecasting Applications: Data Mining Meets Instruction Tuning of Language Models For Multi-modal Time Series Analysis in Low-Resource Settings