Token-choice Mixture-of-Experts (TC-MoE) routes each token to a fixed number of experts, limiting dynamic computation allocation and requiring auxiliary losses to maintain load balance. We propose Expert Threshold (ET) routing, where each expert maintains an exponential moving average (EMA) threshold estimated from the global token distribution. At both training and inference, each token is independently routed to an expert if its score exceeds the expert's threshold, enabling dynamic computation allocation while achieving load balance without auxiliary losses. This fully causal mechanism eliminates dependence on other tokens in the batch, making it well-suited for autoregressive language modeling. In pretraining experiments scaling to 2.4B parameters on FineWeb-Edu, ET achieves 0.067 lower cross-entropy loss than TC-MoE, equivalent to reaching the same performance with 1.6$\times$ fewer tokens.
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Tongji University（同济大学）；Cornell University（康奈尔大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；AI Training Platform Team, Shenzhen Loop Area Institute（深圳环域研究院AI训练平台团队）
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
AGH University of Krakow, Poland（克拉科夫AGH大学）；Faculty of Mathematics（数学系）；Computer Science, Jagiellonian University, Poland（计算机科学系，杰哥利安大学，波兰）；Centre for Credible Artificial Intelligence, Warsaw University of Technology（可信人工智能中心，华沙技术大学）
JevSoup: System-One Routing for Training-Free LoRA Composition
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Malaya（马来亚大学）；Georgian College（乔治亚学院）；Huazhong University of Science and Technology（华中科技大学）
Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routing
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；AiRiA；Tsinghua University（清华大学）；Tencent AI Lab（腾讯AI Lab）
