We present Brainstacks, a modular architecture for continual multi-domain fine-tuning of large language models that packages domain expertise as frozen adapter stacks composing additively on a shared frozen base at inference. Five interlocking components: (1) MoE-LoRA with Shazeer-style noisy top-2 routing across all seven transformer projections under QLoRA 4-bit quantization with rsLoRA scaling; (2) an inner loop performing residual boosting by freezing trained stacks and adding new ones; (3) an outer loop training sequential domain-specific stacks with curriculum-ordered dependencies; (4) null-space projection via randomized SVD constraining new stacks to subspaces orthogonal to prior directions, achieving zero forgetting in isolation; (5) an outcome-based sigmoid meta-router trained on empirically discovered domain-combination targets that selectively weights stacks, enabling cross-domain composition. Two boundary experiments: (6) PSN pretraining on a randomly initialized model; (7) per-domain RL (DPO/GRPO) validating compatibility with post-SFT alignment. Validated on TinyLlama-1.1B (4 domains, 9 stacks) and Gemma 3 12B IT (5 domains, 10 stacks), MoE-LoRA achieves 2.5x faster convergence than parameter-matched single LoRA, residual boosting breaks through the single-stack ceiling, and the routed system recovers generation quality destroyed by ungated stack accumulation. The central finding: the outcome-based router discovers that domain stacks encode transferable cognitive primitives (instruction-following clarity, numerical reasoning, procedural logic, chain-of-thought structure) rather than domain-specific knowledge, with medical prompts routing to chat+math stacks in 97% of cases despite zero medical data in those stacks.
CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Shanghai AI Laboratory(上海人工智能实验室);Beihang University(北京航空航天大学);Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);University of Sydney(悉尼大学)
When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications
City University of Hong Kong(香港城市大学);Jarvis Research Center(贾维斯研究中心);Tencent YouTu Lab(腾讯优图实验室);Southern University of Science and Technology(南方科技大学);University of Science and Technology of China(中国科学技术大学)
PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);Southeast University, Nanjing, China(南京东南大学);Wuhan AI Research, Wuhan, China(武汉人工智能研究院)
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
School of Computer Science and Technology, Beijing Jiaotong University, China(北京交通大学计算机科学与技术学院);Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China(广西trusted软件重点实验室,桂林电子科技大学);Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China(北京交通数据挖掘与具身智能重点实验室);Institute of AI for Industries, Chinese Academy of Sciences, China(工业人工智能研究院,中国科学院);Nanjing Institute of Software Technology, China(南京软件技术研究院)
MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
iFLYTEK Co., Ltd(科大讯飞股份有限公司);University of Pennsylvania(宾夕法尼亚大学);Lenovo Connect Co., Ltd(联想智联有限公司);Niuxin Network Technology Co., Ltd(牛信网络科技有限公司);East China Normal University(华东师范大学)
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment
Harbin Institute of Technology(哈尔滨工业大学);Beijing Normal University - Hong Kong Baptist University United International College(北京师范大学-香港浸会大学联合国际学院);Qilu University of Technology(齐鲁工业大学)
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science(电子电气工程学院,上海工程技术大学);Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University(计算神经科学与脑启发智能重点实验室,复旦大学)
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence(Mohamed bin Zayed大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Kyoto University(京都大学);Tencent AI Lab(腾讯AI实验室);Tomorrow Advancing Life(明天生命科技)
DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
City University of Hong Kong(香港城市大学);Tsinghua University(清华大学);Peking University(北京大学);University of Michigan(密歇根大学);University of Chinese Academy of Sciences(中国科学院大学)
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University(自动化系,清华大学);Academy of Medical Engineering and Translational Medicine, Tianjin University(医学工程与转化医学学院,天津大学)
LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design
Shanghai Jiao Tong University(上海交通大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);National University of Singapore(新加坡国立大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.(浪潮 Genersoft 有限公司);Inspur Group Co. Ltd.(浪潮集团有限公司);Interactive Entertainment Group, Tencent Inc(腾讯公司互动娱乐集团);Nanjing University of Information Science and Technology(南京信息工程大学);Northeastern University(东北大学)
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Beijing Laboratory of Advanced Information Network(北京先进信息网络实验室);Beijing Key Laboratory of Network System Architecture and Convergence(北京网络系统架构与融合重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Department of Electrical and Computer Engineering and Institute for Data Science and Computing(电气与计算机工程系和数据科学与计算研究所);University of Miami(迈阿密大学);Information Systems Technology and Design Pillar(信息系统技术与设计支柱)
GuiLoMo: Allocating Expert Number and Rank for LoRA-MoE via Bilevel Optimization with GuidedSelection Vectors
The University of Hong Kong(香港大学);Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学);University of California, Los Angeles(加州大学洛杉矶分校);National Science Library, Chinese Academy of Sciences(中国科学院国家科学图书馆)
SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
Department of Computer Science, Hong Kong Chu Hai College(香港 Chu Hai 学院计算机科学系);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系)
From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(上海智能信息处理关键实验室,复旦大学计算机学院);Shanghai Collaborative Innovation Center on Intelligent Visual Computing(上海智能视觉计算协同创新中心);Singapore Management University(新加坡管理大学)
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
Beijing Academy of Artificial Intelligence (BAAI)(北京人工智能研究院(BAAI));School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)