Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Electronic and Information, Shanghai Dianji University（上海电机学院电子信息学院）
Chaoxiang Cai, Longrong Yang, Minghe Weng, Xuewei Li, Zequn Qin, Xi Li
The mixture-of-experts (MoE) architecture, which replaces dense networks with sparse ones, has attracted significant attention in large vision-language models (LVLMs) for achieving comparable performance while activating far fewer parameters. Existing MoE architectures for LVLMs primarily focus on token-to-expert routing (TER), encouraging different experts to specialize in processing specific tokens. However, these methods typically rely on the load balancing mechanism, neglecting the inherent distributional differences between vision and language modalities. To address this limitation, we propose the Long-Tailed Distribution-aware Router (LTDR) for vision-language TER, which tackles two key challenges: (1) Modality-specific distribution-aware routing. We observe that language TER generally follows a relatively uniform distribution, whereas vision TER exhibits a long-tailed distribution. This modality discrepancy motivates the design of specialized routing strategies for each modality. (2) Vision-specific dynamic expert activation. Recognizing the importance of high-information vision tail tokens, we introduce a data-augmentation-inspired strategy that increases the number of activated experts, ensuring sufficient learning for these rare but informative tokens. On vision-language and vision benchmarks, our approach achieves consistent improvements, boosting performance by 1.2% / 2.1% on vision-language and 1.6% on vision benchmarks.
We investigate the feasibility of using mixtures of interpretable experts (MoIE) to build interpretable image classifiers on MNIST10. MoIE uses a black-box router to assign each input to one of many inherently interpretable experts, thereby providing insight into why a particular classification decision was made. We find that a naively trained MoIE will learn to 'cheat', whereby the black-box router will solve the classification problem by itself, with each expert simply learning a constant function for one particular class. We propose to solve this problem by introducing interpretable routers…
LPT++: Efficient Training on Mixture of Long-tailed Experts
School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；The Hong Kong Polytechnic University（香港理工大学）
Exposing and Defending the Achilles' Heel of Video Mixture-of-Experts
Nanjing University（南京大学）；China Mobile Information Technology Co., Ltd.（中国移动信息科技有限公司）；Nanjing University of Information Science and Technology（南京信息科技大学）
Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
Korea Advanced Institute of Science and Technology（韩国科学技术院）
Real-world data often have a long-tailed distribution, where the number of samples per class is not equal over training classes. The imbalanced data form a biased feature space, which deteriorates the performance of the recognition model. In this paper, we propose a novel long-tailed recognition method to balance the latent feature space. First, we introduce a MixUp-based data augmentation technique to reduce the bias of the long-tailed data. Furthermore, we propose a new supervised contrastive learning method, named Supervised contrastive learning on Mixed Classes (SMC), for blended images. S…
GEMINUS: Dual-aware Global and Scene-Adaptive Mixture-of-Experts for End-to-End Autonomous Driving
School of Automotive Studies, Tongji University（汽车学院，同济大学）；School of Electronics and Information Engineering, Control Science and Engineering Department, Tongji University（电子信息学院，控制科学与工程系，同济大学）
Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）
Divide, Weight, and Route: Difficulty-Aware Optimization with Dynamic Expert Fusion for Long-tailed Recognition
the MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（信息模式分析与机器智能国家重点实验室）；Collaborative Innovation Center of Novel Software Technology and Industrialization（新型软件技术与产业化协同创新中心）；College of Computer Science and Technology（计算机科学与技术学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Robust Mixture-of-Expert Training for Convolutional Neural Networks
Michigan State University（密歇根州立大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；MIT（麻省理工学院）；Harvard University（哈佛大学）；UC Santa Barbara（加州大学圣巴巴拉分校）；Carnegie Mellon University（卡内基梅隆大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；IBM Research（IBM研究院）
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
