Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppressed probabilities recover. For fixed experts, a stationary-loss barrier identifies the face supporting the optimal routed output. In a common-target square-loss model with freely trainable expert outputs, slow adaptation tracks fixed-expert dynamics on finite horizons, while every fixed positive adaptation rate yields an interior routing limit. An exact normal-velocity decomposition separates complementary-expert covariance from relative gradient advantage. Under positive two-sided power bounds along recovery paths, uniform over initial masses, the resulting coefficient's order of vanishing gives logarithmic or power-law recovery times. Integrations of constructed fields reproduce these orders. An exact discrete log-odds identity relates neural updates to threshold crossing. In paired Transformer-MoE training, removing a class-conditional router-gradient contrast delays confirmed recovery and reduces accumulated log-odds on fixed tokens, both with and without capacity constraints.
How Sparse Probability Maps Shape Mixture-of-Experts Routing
Técnico, Universidade de Lisboa(里斯本大学高等技术学院);INESC-ID(INESC-ID研究所);Instituto de Telecomunicações(电信研究所);ELLIS Unit Lisbon(ELLIS里斯本分部);Gandara AI(Gandara AI公司)
Asymptotic Signal Subspace Recovery in Softmax Attention Models
Faculty of Computer Science and Engineering(计算机科学与工程学院);Ho Chi Minh City University of Technology(胡志明市技术大学);Vietnam National University Ho Chi Minh City(越南胡志明市国家大学);VNU-HCM
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Institute of Advanced Intelligence and Computing (IAIC), A*STAR(先进智能与计算研究所(IAIC),新加坡科技研究局);Nanyang Technological University(南洋理工大学);ST Engineering Geo-Insights(新科工程地理信息公司);Sichuan University(四川大学);Shanghai Jiao Tong University(上海交通大学);University of Science and Technology of China(中国科学技术大学)
Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models
School of Computing, National University of Singapore, Singapore.(新加坡国立大学计算机学院);ARC Centre of Excellence for the Mathematical Analysis of Cellular Systems(细胞系统数学分析卓越研究中心);School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia.(昆士兰科技大学数学科学学院);Department of Statistics and Data Science, University of Texas at Austin, Austin, USA(德克萨斯大学奥斯汀分校统计与数据科学系)
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
Department of Computer Science and Software Engineering(计算机科学与软件工程系);Auburn University(阿伯拉罕大学);Department of Information Management(信息管理系);National Central University(国立中央大学)
GRIP: Algorithm-Agnostic Machine Unlearning for Mixture-of-Experts via Geometric Router Constraints
School of Computer Science(计算机科学学院);Georgia Institute of Technology(佐治亚理工学院);Department of Electrical Engineering(电气工程系);Tsinghua University(清华大学);School of Electrical and Computer Engineering(电气与计算机工程学院)
Routing Recovery for UAV Networks with Deliberate Attacks: A Reinforcement Learning based Approach
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Xidian University(西安电子科技大学);China Academy of Railway Sciences Corporation Limited(中国铁道科学研究院集团有限公司)
PolicyAttention: Softmax Attention Implements Policy Mirror Descent for Closed-Loop Control
Quantitative Research Society(量化研究学会);Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The University of Hong Kong(香港大学)