Reliable channel estimation (CE) is fundamental for robust communication in dynamic wireless environments, where models must generalize across varying conditions such as signal-to-noise ratios (SNRs), the number of resource blocks (RBs), and channel profiles. Traditional deep learning (DL)-based methods struggle to generalize effectively across such diverse settings, particularly under multitask and zero-shot scenarios. In this work, we propose MoE-CE, a flexible mixture-of-experts (MoE) framework designed to enhance the generalization capability of DL-based CE methods. MoE-CE provides an appropriate inductive bias by leveraging multiple expert subnetworks, each specialized in distinct channel characteristics, and a learned router that dynamically selects the most relevant experts per input. This architecture enhances model capacity and adaptability without a proportional rise in computational cost while being agnostic to the choice of the backbone model and the learning algorithm. Through extensive experiments on synthetic datasets generated under diverse SNRs, RB numbers, and channel profiles, including multitask and zero-shot evaluations, we demonstrate that MoE-CE consistently outperforms conventional DL approaches, achieving significant performance gains while maintaining efficiency.
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）
S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；School of Integrated Circuits, Peking University（北京大学集成电路学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）
ECG-MoE: Mixture-of-Expert Electrocardiogram Foundation Model
Department of Computer Science（计算机科学系）；Emory University（埃默里大学）；School of Computer Science（计算机科学学院）；University of Oklahoma（俄克拉荷马大学）；Center of Data Science（数据科学中心）
Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in scaling up deep neural networks to an extreme scale. Despite that numerous efforts have been made to improve the performance of MoE from the model design or system optimization perspective, existing MoE dispatch patterns are still not able to fully exploit the underlying heterogeneous network environments. In this paper, we propose TA-MoE, a topology-aware routing strategy for large-scale MoE trainging, from a model-system co-design perspective, which can dynamically adjust the MoE dispatch pattern according to the ne…
University of Oxford（牛津大学）；Mila - Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；Google DeepMind（谷歌DeepMind）
MoE-Loco: Mixture of Experts for Multitask Locomotion
IIIS, Tsinghua University, Beijing, China（清华大学信息学院，北京，中国）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）；Massachusetts Institute of Technology, MA, USA（麻省理工学院，马萨诸塞州，美国）
MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
As the training of giant dense models hits the boundary on the availability and capability of the hardware resources today, Mixture-of-Experts (MoE) models become one of the most promising model architectures due to their significant training cost reduction compared to a quality-equivalent dense model. Its training cost saving is demonstrated from encoder-decoder models (prior works) to a 5x saving for auto-aggressive language models (this work along with parallel explorations). However, due to the much larger model size and unique architecture, how to provide fast MoE model inference remains …
Beihang University（北京航空航天大学）；The Institute of Acoustics of the Chinese Academy of Sciences（中国科学院声学研究所）；Microsoft（微软）
The sparse Mixture-of-Experts (MoE) model is powerful for large-scale pre-training and has achieved promising results due to its model capacity. However, with trillions of parameters, MoE is hard to be deployed on cloud or mobile environment. The inference of MoE requires expert parallelism, which is not hardware-friendly and communication expensive. Especially for resource-limited downstream tasks, such sparse structure has to sacrifice a lot of computing efficiency for limited performance gains. In this work, we observe most experts contribute scarcely little to the MoE fine-tuning and infer…
College of Cyber Security, Jinan University（暨南大学网络空间安全学院）；College of Mathematics and Computer Science, Shantou University（汕头大学数学与计算机科学学院）；School of Information Engineering, Guangdong Eco-Engineering Polytechnic（广东生态工程职业学院信息工程学院）；Department of Computer Science, University of Illinois Chicago（芝加哥伊利诺伊大学计算机科学系）
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
University of Warsaw（华沙大学）；Wroclaw University of Science and Technology（弗罗茨瓦夫理工大学）；Institute of Mathematics, Polish Academy of Sciences（波兰科学院数学研究所）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University of Science and Technology（南京理工大学）；AIRIA
Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing
School of Informatics, Xiamen University（厦门大学信息学院）；School of Electronic Science and Engineering, Xiamen University（厦门大学电子科学与技术学院）；Kuaishou Technology Co., Ltd（快手科技有限公司）
MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Peng Cheng Laboratory（鹏城实验室）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Kunlun 2050 Research（昆仑2050研究院）；Skywork AI（智源天工AI公司）
AW-MoE: All-Weather Mixture of Experts for Robust Multi-Modal 3D Object Detection
Fujian Key Laboratory of Urban Intelligent Sensing and Computing（福建城市智能感知与计算重点实验室）；School of Informatics, Xiamen University（厦门大学信息学院）；Zhongguancun Academy（中关村学院）
We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynamic routing in MoE layers to satisfy the constraints of existing software and hardware. These formulations force a tradeoff between model quality and hardware efficiency, as users must choose between dropping tokens from the computation or wasting computation and memory on padding. To address these limitations, we reformulate MoE computation in terms of block-sparse operations and develop new block-sparse GPU kernels t…
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Singidunum University（辛吉杜努姆大学）；Google（谷歌）；University of Belgrade（贝尔格莱德大学）；Google Brain（谷歌大脑）
Rapid advancements in deep learning have led to many recent breakthroughs. While deep learning models achieve superior performance, often statistically better than humans, their adoption into safety-critical settings, such as healthcare or self-driving cars is hindered by their inability to provide safety guarantees or to expose the inner workings of the model in a human understandable form. We present MoËT, a novel model based on Mixture of Experts, consisting of decision tree experts and a generalized linear model gating function. Thanks to such gating function the model is more expressive t…
Survival analysis is a task to model the time until an event of interest occurs, widely used in clinical and biomedical research. A key challenge is to model patient heterogeneity while also adapting risk predictions to both individual characteristics and temporal dynamics. We propose a dual mixture-of-experts (MoE) framework for discrete-time survival analysis. Our approach combines a feature-encoder MoE for subgroup-aware representation learning with a hazard MoE that leverages patient features and time embeddings to capture temporal dynamics. This dual-MoE design flexibly integrates with ex…
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology（国防科技大学并行与分布计算国家重点实验室，计算机科学与技术学院）
The sparsely-gated Mixture of Experts (MoE) can magnify a network capacity with a little computational complexity. In this work, we investigate how multi-lingual Automatic Speech Recognition (ASR) networks can be scaled up with a simple routing algorithm in order to achieve better accuracy. More specifically, we apply the sparsely-gated MoE technique to two types of networks: Sequence-to-Sequence Transformer (S2S-T) and Transformer Transducer (T-T). We demonstrate through a set of ASR experiments on multiple language data that the MoE networks can reduce the relative word error rates by 16.3% …
Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using $\sim$4 times less compute. This gap narrows at scale, but o…
