We present a text-reconstruction attack on mixture-of-experts (MoE) language models that recovers tokens from expert selections alone. In MoE models, each token is routed to a subset of expert subnetworks; we show these routing decisions leak substantially more information than previously understood. Prior work using logistic regression achieves limited reconstruction; we show that a 3-layer MLP improves this to 63.1% top-1 accuracy, and that a transformer-based sequence decoder recovers 91.2% of tokens top-1 (94.8% top-10) on 32-token sequences from OpenWebText after training on 100M tokens. These results connect MoE routing to the broader literature on embedding inversion. We outline practical leakage scenarios (e.g., distributed inference and side channels) and show that adding noise reduces but does not eliminate reconstruction. Our findings suggest that expert selections in MoE deployments should be treated as sensitive as the underlying text.
MoE$^2$-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);KTH Royal Institute of Technology(瑞典皇家理工学院);University of Science and Technology of China(中国科学技术大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Soochow University(苏州大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
The Chinese University of Hong Kong(香港中文大学);Renmin University of China(中国人民大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);Westlake University(西湖大学)
Multi-level context Modeling for consistent expert selection in Mixture-of-Experts
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);School of AI, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)人工智能学院)
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);Chinese University of Hong Kong(中国香港大学)
TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Acceleration
Institute for Artificial Intelligence(人工智能研究院);School of Integrated Circuits(集成电路学院);Yuanpei College(元培学院);Beijing Advanced Innovation Center for Integrated Circuits(北京集成电路先进创新中心)
What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models
Guangdong University of Technology(广东工业大学);Soochow University(苏州大学);University of Copenhagen(哥本哈根大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology(国防科技大学并行与分布计算国家重点实验室,计算机科学与技术学院)
ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
The Department of Electrical and Computer Engineering, The University of Hong Kong(香港大学电子与计算机工程系);The School of Integrated Circuit Science and Engineering, Beihang University(北航集成电路科学与工程学院)
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
University of Warsaw(华沙大学);Wroclaw University of Science and Technology(弗罗茨瓦夫理工大学);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所);Institute of Fundamental Technological Research, Polish Academy of Sciences(波兰科学院基础技术研究所)
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)