Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
Shenzhen Campus of Sun Yat-sen University, China(中山大学深圳校区);Northeastern University China(东北大学);Shenzhen Campus of Sun Yat-sen University China(中山大学深圳校区);Nanyang Technological University Singapore(南洋理工大学);Northeastern University(东北大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Nanyang Technological University(南洋理工大学);Institute for Clarity in Documentation Dublin Ohio USA(文档清晰研究所);Inria Paris-Rocquencourt Rocquencourt France(巴黎-罗quentourt研究所);Rajiv Gandhi University Doimukh Arunachal Pradesh India(拉贾·甘地大学);Tsinghua University Haidian Qu Beijing Shi China(清华大学);Palmer Research Laboratories San Antonio Texas USA(帕勒研究中心);Institute for Clarity in Documentation(文档清晰研究所);Inria Paris-Rocquencourt(巴黎-罗quentourt研究所);Rajiv Gandhi University(拉贾·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕勒研究中心)
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, Dacheng Tao
Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and more than ten machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications.
AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University(计算机科学与技术系,人工智能研究院,清华大学);Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学);State Key Laboratory of Multimedia Information Processing, Peking University(多媒体信息处理国家重点实验室,北京大学);School of Software Microelectronics, Peking University(软件微电子学院,北京大学);Institute of Intelligent Computing, Alibaba Group(智能计算研究院,阿里巴巴集团);Shanghai Artificial Intelligence Laboratory, Shanghai, China(上海人工智能实验室,上海,中国);Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China(江苏省语言能力协同创新中心,江苏,中国);ModelTC Open Source Organization, Beijing, China(ModelTC开源组织,北京,中国)
RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness
MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所MAIS);Centre for Artificial Intelligence and Robotics, HKISI-CAS(香港科学院智能机器人中心);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区);Shenzhen Loop Area Institute(深圳河套学院);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机科学学院多媒体信息处理国家重点实验室)
PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Data Science and Artificial Intelligence Research Institute, China United Network Communications Group Co., Ltd.(中国联合网络通信集团有限公司数据科学与人工智能研究院)
Model Composition for Multimodal Large Language Models
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Institute of Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University(浙江大学 ZIP 实验室);South China University of Technology(华南理工大学);Pazhou Lab(琶洲实验室);Key Laboratory of Big Data and Intelligent Robot, Ministry of Education(大数据与智能机器人教育部重点实验室)
MAny: Merge Anything for Multimodal Continual Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology(国防科技大学计算机科学与技术学院);National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology(国防科技大学并行与分布式计算国家重点实验室);State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);School of Computer Science, Tsinghua University(清华大学计算机科学与技术系)
Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology(南方科技大学可信自主系统研究院与计算机科学与工程系);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
AIML, University of Adelaide(AIML,阿德莱德大学);Monash University(墨尔本大学);Australian National University(澳大利亚国立大学);Tiktok, Australia(Tiktok,澳大利亚);Zhejiang University(浙江大学)
Towards Efficient Multimodal Unified Reasoning Model via Model Merging
Beijing University of Posts and Telecommunications(北京邮电大学);Nanyang Technological University(南洋理工大学);Beijing Key Laboratory of Network System and Network Culture(北京网络系统与网络文化重点实验室);Key Laboratory of Interactive Technology and Experience System, Ministry of Culture and Tourism(文化和旅游部交互技术与体验系统重点实验室);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国)
Enhancing Instruction-Following Capability of Visual-Language Models by Reducing Image Redundancy
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Kuaishou Technology(快手科技);CAIR, HKISI, Chinese Academy of Sciences(中国科学院香港创新研究院人工智能与机器人创新中心)
A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
JTB Technology Corp.(JTB技术公司);Stockton University(斯托顿大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);AppCubic USA(AppCubic美国公司);Nomad Sustaintech LTD(Nomad可持续科技有限公司);Georgia Institute of Technology(佐治亚理工学院);Emory University(埃默里大学);University of Liverpool(利物浦大学);Indiana University(印第安纳大学);Purdue University(普渡大学)
Can a Large Language Model Assess Urban Design Quality? Evaluating Walkability Metrics Across Expertise Levels
Singapore-ETH Centre, Future Cities Lab Global Programme, CREATE campus(新加坡-ETH中心,未来城市实验室全球计划,CREATE校区);National University of Singapore(新加坡国立大学);Takenaka Corporation(Takenaka公司);Department of Architecture, National University of Singapore(新加坡国立大学建筑系)
The Evolution of Video Anomaly Detection: A Unified Framework from DNN to MLLM
Beijing Jiaotong University(北京交通大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Zhongguancun Academy, Beijing, China(中关村学院,北京,中国)
AdaTok: Adaptive Token Compression with Object-Aware Representations for Efficient Multimodal LLMs
Institute of Medical Technology, Peking University Health Science Center(北京大学医学部医学技术研究所);Department of Biomedical Engineering, Peking University(北京大学生物医学工程系);National Biomedical Imaging Center, Peking University(北京大学国家生物医学成像中心)
Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education
Technical University of Munich(慕尼黑工业大学);TUM School of Social Sciences and Technology(慕尼黑工业大学社会科学与技术学院);University of Georgia(佐治亚大学);AI4STEM Education Center(AI4STEM教育中心)