Recent advancements in multimodal large language models (MLLMs) have demonstrated considerable potential for comprehensive 3D scene understanding. However, existing approaches typically utilize only one or a limited subset of 3D modalities, resulting in incomplete representations of 3D scenes and reduced interpretive accuracy. Furthermore, different types of queries inherently depend on distinct modalities, indicating that uniform processing of all modality tokens may fail to effectively capture query-specific context. To address these challenges, we propose Uni3D-MoE, a sparse Mixture-of-Experts (MoE)-based 3D MLLM designed to enable adaptive 3D multimodal fusion. Specifically, Uni3D-MoE integrates a comprehensive set of 3D modalities, including multi-view RGB and depth images, bird's-eye-view (BEV) maps, point clouds, and voxel representations. At its core, our framework employs a learnable routing mechanism within the sparse MoE-based large language model, dynamically selecting appropriate experts at the token level. Each expert specializes in processing multimodal tokens based on learned modality preferences, thus facilitating flexible collaboration tailored to diverse task-specific requirements. Extensive evaluations on standard 3D scene understanding benchmarks and specialized datasets demonstrate the efficacy of Uni3D-MoE.
BIG-MoE: Bypass Isolated Gating MoE for Generalized Multimodal Face Anti-Spoofing
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Great Bay University（大湾区大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学大数据系统计算技术国家工程实验室）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Microsoft（微软）
Mixture of Experts (MoE) is able to scale up vision transformers effectively. However, it requires prohibiting computation resources to train a large MoE transformer. In this paper, we propose Residual Mixture of Experts (RMoE), an efficient training pipeline for MoE vision transformers on downstream tasks, such as segmentation and detection. RMoE achieves comparable results with the upper-bound MoE training, while only introducing minor additional training cost than the lower-bound non-MoE training pipelines. The efficiency is supported by our key observation: the weights of an MoE transforme…
MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
National University of Defense Technology（国防科技大学）；Shanghai AI Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；ShanghaiTech University（上海科技大学）
GM-MoE: Low-Light Enhancement with Gated-Mechanism Mixture-of-Experts
Xinjiang University（新疆大学）；Harbin University of Commerce（哈尔滨商业大学）；Changchun University of Science and Technology（长春理工大学）；University of Science and Technology Beijing（北京科技大学）；University of Trento（特伦托大学）
DynFS-MoE: Dynamic Functional-Structural Mixture-of-Experts for Post-Traumatic Epilepsy Diagnosis
Department of Systems Engineering, Stevens Institute of Technology（史蒂文斯理工学院系统工程系）；Department of Neurosurgery, Robert Wood Johnson Medical School, Rutgers University（罗格斯大学罗伯特·伍德·约翰逊医学院神经外科）
AW-MoE: All-Weather Mixture of Experts for Robust Multi-Modal 3D Object Detection
Fujian Key Laboratory of Urban Intelligent Sensing and Computing（福建城市智能感知与计算重点实验室）；School of Informatics, Xiamen University（厦门大学信息学院）；Zhongguancun Academy（中关村学院）
Sparsely-gated Mixture of Experts networks (MoEs) have demonstrated excellent scalability in Natural Language Processing. In Computer Vision, however, almost all performant networks are "dense", that is, every input is processed by every parameter. We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks. When applied to image recognition, V-MoE matches the performance of state-of-the-art networks, while requiring as little as half of the compute at inference time. Further, we propose an extension to the routi…
Stable Routing for Mixture-of-Experts in Class-Incremental Learning
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
