We introduce MoMa, a novel modality-aware mixture-of-experts (MoE) architecture designed for pre-training mixed-modal, early-fusion language models. MoMa processes images and text in arbitrary sequences by dividing expert modules into modality-specific groups. These groups exclusively process designated tokens while employing learned routing within each group to maintain semantically informed adaptivity. Our empirical results reveal substantial pre-training efficiency gains through this modality-specific parameter allocation. Under a 1-trillion-token training budget, the MoMa 1.4B model, featuring 4 text experts and 4 image experts, achieves impressive FLOPs savings: 3.7x overall, with 2.6x for text and 5.2x for image processing compared to a compute-equivalent dense baseline, measured by pre-training loss. This outperforms the standard expert-choice MoE with 8 mixed-modal experts, which achieves 3x overall FLOPs savings (3x for text, 2.8x for image). Combining MoMa with mixture-of-depths (MoD) further improves pre-training FLOPs savings to 4.2x overall (text: 3.4x, image: 5.3x), although this combination hurts performance in causal inference due to increased sensitivity to router accuracy. These results demonstrate MoMa's potential to significantly advance the efficiency of mixed-modal, early-fusion language model pre-training, paving the way for more resource-efficient and capable multimodal AI systems.
ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
NepAl Applied Mathematics and Informatics Institute（尼泊尔应用数学与信息学研究所）；Fogsphere (Redev.AI Ltd)（福格球（Redev.AI有限公司））；University of Lausanne（洛桑大学）；West Virginia University（西弗吉尼亚大学）；University of Verona（维罗纳大学）；University College London（伦敦大学学院）；University of Aberdeen（阿伯丁大学）
Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Ant Group Joint Lab of Knowledge Graph（浙江大学-蚂蚁集团知识图谱联合实验室）；Ant Group（蚂蚁集团）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Big Data Intelligent Computing（浙江省大数据智能计算重点实验室）
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
School of Software, Tsinghua University, Beijing, China（清华大学软件学院，北京，中国）；TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China（天津大学计算机科学与技术学院，中国）；School of New Media and Communication, Tianjin University, China（天津大学新媒体与传播学院，中国）
CAME-AB: Cross-Modality Attention with Mixture-of-Experts for Antibody Binding Site Prediction
Generative AI Research and Development Center The Hong Kong University of Science and Technology（生成式人工智能研究与发展中心 香港科学大学）；MILES The University of Hong Kong（MILES 香港大学）；College of Veterinary Medicine Jilin University（吉林大学兽医学院）；School of Chemistry and Chemical Engineering South China University of Technology（华南理工大学化学与化工学院）；School of Biomedical Sciences The University of Hong Kong（香港大学生物医学科学学院）；Computational Immunology Centre BayVax Biotech Limited（计算免疫学中心 BayVax 生物技术有限公司）；College of Electronic and Information Engineering Southwest University（西南大学电子与信息工程学院）
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）
The University of Edinburgh（爱丁堡大学）；Imperial College, London（伦敦帝国学院）；University of Surrey（萨里大学）；Queen Mary University of London（伦敦玛丽女王大学）；Microsoft Research, Cambridge（微软研究院剑桥分部）
Generalist-Specialist Mixture-of-Experts for Rare Pathology Detection in Multimodal Imaging
Technical University of Munich (TUM)（慕尼黑工业大学）；TUM University Hospital（慕尼黑工业大学附属医院）；Hasso Plattner Institute for Digital Engineering（哈索·普拉特纳数字工程研究所）；University of Potsdam（波茨坦大学）；Imperial College London（伦敦帝国理工学院）
Addressing Missing and Noisy Modalities in One Solution: Unified Modality-Quality Framework for Low-quality Multimodal Data
Department of Computer Science（计算机科学系）；South China Normal University（华南师范大学）；School of Electronics and Information Technology（电子与信息学院）；Sun Yat-sen University（中山大学）
AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba
Pengcheng Laboratory（鹏城实验室）；Shaanxi University of Science & Technology（陕西科技大学）；School of Computer Science（计算机科学学院）；Northwestern Polytechnical University（西北工业大学）
Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation
University of Science and Technology of China（中国科学技术大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）人工智能学域）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology（香港科技大学计算机科学与工程学系）
BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration
Department of Computer Science, Virginia Tech, Blacksburg, VA, USA（计算机科学系，弗吉尼亚理工学院，布莱克斯堡，VA，美国）；Department of Agricultural and Applied Economics, Virginia Tech, Blacksburg, VA, USA（农业与应用经济学系，弗吉尼亚理工学院，布莱克斯堡，VA，美国）
