We resolve the long-standing "impossible tuning" issue for the classic expert problem and show that, it is in fact possible to achieve regret $O\left(\sqrt{(\ln d)\sum_t \ell_{t,i}^2}\right)$ simultaneously for all expert $i$ in a $T$-round $d$-expert problem where $\ell_{t,i}$ is the loss for expert $i$ in round $t$. Our algorithm is based on the Mirror Descent framework with a correction term and a weighted entropy regularizer. While natural, the algorithm has not been studied before and requires a careful analysis. We also generalize the bound to $O\left(\sqrt{(\ln d)\sum_t (\ell_{t,i}-m_{t,i})^2}\right)$ for any prediction vector $m_t$ that the learner receives, and recover or improve many existing results by choosing different $m_t$. Furthermore, we use the same framework to create a master algorithm that combines a set of base algorithms and learns the best one with little overhead. The new guarantee of our master allows us to derive many new results for both the expert problem and more generally Online Linear Optimization.
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
Department of Computer Science and Engineering, Korea University, Republic of Korea（韩国大学计算机科学与工程系）；KT, Republic of Korea（韩国KT）；School of Software, Soongsil University, Republic of Korea（顺天大学软件学院）
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China, China（苏州先进研究院，中国科学技术大学）
Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）
UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Models
School of Computer Science and Statistics, Trinity College Dublin, Ireland（特里尼蒂学院都柏林分校计算机科学与统计学学院）；School of Computer Science, University College Dublin, Ireland（都柏林大学学院计算机科学学院）
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Columbia University（哥伦比亚大学）；Data Science Institute, Columbia University（哥伦比亚大学数据科学研究所）；Department of Computer Science, Columbia University（哥伦比亚大学计算机科学系）
CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；University of Science and Technology Beijing（北京科技大学）
A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（中国科学院计算技术研究所）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）
Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts
City University of Hong Kong（香港城市大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Oxford（牛津大学）；Harvard University（哈佛大学）；Nanyang Technological University（南洋理工大学）
TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
Tennessee Tech University, \ Alamos National Laboratory Cookeville Tennessee USA；Los Alamos National Laboratory Los Alamos New Mexico USA；Tennessee Tech University Cookeville Tennessee USA；Tennessee Tech University, \ Alamos National Laboratory；Los Alamos National Laboratory；Tennessee Tech University
Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS)（深圳市人工智能与机器人研究院）；Zhejiang University（浙江大学）；Westlake University（西湖大学）；SII；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；Tencent AI Lab（腾讯AI Lab）
MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
Rochester Institute of Technology（罗切斯特理工学院）；University of Rochester（罗切斯特大学）；Meta AI；Rice University（莱斯大学）；Case Western Reserve University（凯斯西储大学）；U.S. DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）；University of Missouri - Kansas City（密苏里大学堪萨斯城分校）；UC Davis（加州大学戴维斯分校）
MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning
School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；Artificial Intelligence Research Institute, iFLYTEK Company Ltd.（iFLYTEK公司人工智能研究院）
