Selecting the best data mixture is critical for successful Supervised Fine-Tuning (SFT) of Multimodal Large Language Models. However, determining the optimal mixture weights across multiple domain-specific datasets remains a significant bottleneck due to the combinatorial search space and the high cost associated with even a single training run. This is the so-called Data Mixture Optimization (DMO) problem. On the other hand, model merging unifies domain-specific experts through parameter interpolation. This strategy is efficient, as it requires only a single training run per domain, yet it often leads to suboptimal models. In this work, we take the best of both worlds, studying model merging as an efficient strategy for estimating the performance of different data mixtures. We train domain-specific multimodal experts and evaluate their weighted parameter-space combinations to estimate the efficacy of corresponding data mixtures. We conduct extensive experiments on 14 multimodal benchmarks, and empirically demonstrate that the merged proxy models consistently select near-optimal mixtures. This decouples the search for optimal mixtures from the resource-intensive training process, thereby providing a scalable and efficient strategy for navigating the complex landscape of mixture weights. Code is publicly available at https://github.com/BerasiDavide/mLLMs_merging_4_DMO.
Unlocking the Potential of Model Merging for Low-Resource Languages
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Center for Data Science, Peking University(北京大学数据科学中心);College of Engineering, Peking University(北京大学工学院)
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
NLP Team, Xiaohongshu Inc., Shanghai, China(小红书自然语言处理团队,小红书公司,上海,中国);Tsinghua University, Beijing, China(清华大学,北京,中国);The University of Tokyo, Tokyo, Japan(东京大学,东京,日本)
The University of Edinburgh(爱丁堡大学);Imperial College, London(伦敦帝国学院);University of Surrey(萨里大学);Queen Mary University of London(伦敦玛丽女王大学);Microsoft Research, Cambridge(微软研究院剑桥分部)
SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture
Inspur Genersoft Co. Ltd.(浪潮 Genersoft 有限公司);Inspur Group Co. Ltd.(浪潮集团有限公司);Interactive Entertainment Group, Tencent Inc(腾讯公司互动娱乐集团);Nanjing University of Information Science and Technology(南京信息工程大学);Northeastern University(东北大学)
ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns
Zhejiang University(浙江大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);Fudan University(复旦大学);Chinese University of Hong Kong(中国香港大学)
Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Polytechnique Montréal(蒙特利尔综合理工学院);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);AI Center, Samsung, South Korea(三星人工智能中心,韩国);Samsung AI Lab, Montreal(三星人工智能实验室,蒙特利尔)
FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts
Department of Automation, Tsinghua University(自动化系,清华大学);Academy of Medical Engineering and Translational Medicine, Tianjin University(医学工程与转化医学学院,天津大学)
Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMs
Zhejiang University(浙江大学);Shanghai Innovation Institute(上海创新研究院);Southeast University(东南大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Google DeepMind(谷歌DeepMind);University of Oxford(牛津大学);McGill University(麦吉尔大学)
Evolutionary Soups: Evolving Mixture-of-Experts for Multi-Objective LLM Alignment
Fraunhofer Institute for Applied Information Technology FIT(弗劳恩霍夫应用信息技术研究所FIT);University of Cologne(科隆大学);University of Stuttgart(斯图加特大学);University of Southampton(南安普顿大学);Soochow University(苏州大学);University Hospital of Cologne(科隆大学医院)
Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)
Expert Merging in Sparse Mixture of Experts with Nash Bargaining
Department of Mathematics, National University of Singapore(新加坡国立大学数学系);Viettel AI, Viettel Group(越南电信AI部门);Faculty of Mathematics and Informatics, Hanoi University of Science and Technology(河内科学技术大学数学与信息学系);Bar Ilan University, Israel(以色列巴伊兰大学);AI Imaging Team, Data Solution Department, FPT Software Japan(日本FPT软件数据解决方案部门AI成像团队)
Unlocking Cross-Scenario Physical Layer Security: A Mixture-of-Experts Framework with Generative Diffusion Models
Shenzhen Research Institute of Northwestern Polytechnical University(西北工业大学深圳研究院);Northwestern Polytechnical University(西北工业大学);Pengcheng Laboratory(鹏城实验室);University of Houston(休斯顿大学)
Merging Multi-Task Models via Weight-Ensembling Mixture of Experts
National Engineering Research Center for Multimedia Software(国家多媒体软件工程技术研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院);Hubei Luojia Laboratory(湖北珞珈实验室);Sun Yat-sen University(中山大学);JD Explore Academy(京东探索研究院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Nanyang Technological University(南洋理工大学)
Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
National Tsing Hua University(国立清华大学);National Taiwan University(国立台湾大学);University of Toronto(多伦多大学);NVIDIA AI Technology Center (NVAITC)(NVIDIA AI技术中心);Samsung Research America(三星美国研究院)
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Hong Kong University of Science and Technology(香港科技大学);Mila - Québec AI Institute(魁北克AI研究院);Université de Montréal(蒙特利尔大学);Fudan University(复旦大学);City University of Hong Kong(香港城市大学)
Hierarchical clustering with maximum density paths and mixture models
Institute of Computer Science and Campus Institute Data Science, University of Göttingen(计算机科学研究所和校园数据科学研究所,哥廷根大学);Max Planck Institute for Dynamics and Self-Organization(动态与自组织Max Planck研究所)
School of Computing and Information Technology, Great Bay University(大湾大学计算与信息学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件工程学院);Department of Computer Science, University of California, Los Angeles(加州大学洛杉矶分校计算机科学系);Department of Computer and Information Science, University of Macau(澳门大学计算机与信息科学系)
MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion
Department of Computing, The Hong Kong Polytechnic University(计算系,香港理工大学);Research Institute of Multiple Agents and Embodied Intelligence, Pengcheng Laboratory(多智能体与具身智能研究院,鹏城实验室)
Efficient Training of Large-Scale AI Models Through Federated Mixture-of-Experts: A System-Level Approach
Division of Electrical and Computer Engineering, Louisiana State University(电气与计算机工程系,路易斯安那州立大学);Division of Computer Science and Engineering, Louisiana State University(计算机科学与工程系,路易斯安那州立大学);Department of Data Science, New Jersey Institute of Technology(数据科学系,新泽西理工学院);Department of Electrical and Computer Engineering, University of Louisiana at Lafayette(电气与计算机工程系,路易斯安那大学拉斐特分校);Department of Electrical and Electronic Engineering, Imperial College London(电气与电子工程系,伦敦帝国理工学院)