arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.06552cs.LG
收藏

细粒度模型合并 via 模块化专家重组

Fine-Grained Model Merging via Modular Expert Recombination

Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(数据科学与人工智能系,香港理工大学);College of Computer Science, Chongqing University(计算机科学学院,重庆大学)

Haiyun Qiu, Xingyu Wu, Liang Feng, Kay Chen Tan

AI总结:

MERGE通过模块化专家重组实现细粒度模型合并,提升模型适应性和存储效率。

摘要:

模型合并通过整合任务特定模型构建多功能模型,而无需标注数据或昂贵的联合再训练。尽管最近的方法通过为每个实例生成定制的合并模型来提高对异构任务的适应性,但它们面临两个关键限制。首先,实例特定的合并模型缺乏可重用性,限制了高质量合并配置和高效批量推理的利用。其次,这些方法将每个任务特定模型视为整体,忽略了诸如注意力和多层感知机层等同源组件的多样化合并性以及不同组件的合并敏感性差异。为了解决这些限制,我们提出了MERGE(模块化专家重组用于细粒度合并),一种能够实现组件级模型合并和输入感知、按需模块重组的方法。MERGE将组件级合并制定为一个双目标优化问题,平衡跨任务性能和存储效率,并开发了一种代理辅助的进化算法以高效地识别帕累托最优的合并配置。这些高质量的配置奠定了一个可重用的模块化专家库,从中一个轻量级的路由网络动态激活和重组模块化专家,以组装输入特定的模型,并在存储限制下实现高效的推理。在各种模型规模、任务类型和微调策略上的广泛实验表明,MERGE在各种情况下均优于强大的基线,并且具有良好的泛化能力。

英文摘要:

Model merging constructs versatile models by integrating task-specific models without requiring labeled data or expensive joint retraining. Although recent methods improve adaptability to heterogeneous tasks by generating customized merged models for each instance, they face two critical limitations. First, the instance-specific merged models lack reusability, restricting the exploitation of high-quality merging configurations and efficient batch inference. Second, these methods treat each task-specific model as a monolithic whole, overlooking the diverse mergeability of homologous components such as attention and multilayer perceptron layers, and the differing merging sensitivities across components. To address these limitations, we propose MERGE (\underline{M}odular \underline{E}xpert \underline{R}ecombination for fine-\underline{G}rained m\underline{E}rging), a method that enables component-wise model merging and input-aware, on-demand module recombination at inference. MERGE formulates component-wise merging as a bi-objective optimization problem that balances cross-task performance and storage efficiency, and develops a surrogate-assisted evolutionary algorithm to efficiently identify Pareto-optimal merging configurations. These high-quality configurations underpin a reusable modular expert library, from which a lightweight routing network dynamically activates and recombines modular experts to assemble input-specific models and enable efficient inference under storage constraints. Extensive experiments across various model scales, task types, and fine-tuning strategies demonstrate that MERGE consistently outperforms strong baselines and generalizes effectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑