SMAT：简单高效的合并感知训练
SMAT: Simple and Efficient Merge-Aware Training
浏览论文内容
中文总结 AI 辅助
SMAT通过模拟合并操作联合优化专家损失与期望损失，实现高效合并感知训练，在四个骨干网络上平均提升1.07-2.16分，训练开销低于2%。
中文摘要 AI 辅助
模型合并无需联合重训练即可整合多个专家的能力，但标准专家训练仅优化任务损失，无法保证合并后的良好性能。合并感知训练（MAT）旨在提升合并性能，但现有方法未能充分考虑常见的合并操作，且增加了训练成本。我们观察到，从专家的角度来看，常见合并方法可由三种操作描述：缩放（Scale）对其自身更新进行重新加权，掩码（Mask）移除选定坐标，扰动（Perturb）添加来自其他专家的更新。基于这一视角，我们提出SMAT（简单MAT），它联合优化专家损失和通过采样缩放系数、掩码和加性噪声生成的模拟合并参数处的期望损失。我们进一步引入周期调度、内核融合和参数存储切换，使SMAT高效，每步仅需一次前向和一次反向传播。在四个语言和视觉-语言骨干网络上，SMAT在五种合并方法上的平均得分比每个骨干网络的最强基线提高1.07-2.16分，训练时间开销比标准微调低2%以内。
英文摘要
Model merging integrates the capabilities of multiple experts without joint retraining, but standard expert training optimizes task loss alone and does not guarantee good performance after merging. Merge-aware training (MAT) aims to improve merged performance, but existing methods do not fully account for common merging operations and add training cost. We observe that, from an expert's perspective, common merging methods can be described by three operations: Scale reweights its own update, Mask removes selected coordinates, and Perturb adds updates from other experts. Based on this view, we introduce SMAT (Simple MAT), which jointly optimizes expert loss and expected loss at simulated merged parameters generated by sampling scaling coefficients, masks, and additive noise. We further introduce periodic scheduling, kernel fusion, and parameter storage switching to make SMAT efficient, with one forward and one backward pass per step. Across four language and vision-language backbones, SMAT improves the mean score across five merging methods by 1.07-2.16 points over the strongest baseline for each backbone, with less than 2% training-time overhead over standard fine-tuning.
发表机构
- The Hong Kong Polytechnic University (PolyU)（香港理工大学）
- The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
- The Chinese University of Hong Kong（香港中文大学）
- PolyU-Daya Bay Technology and Innovation Research Institute（香港理工大学-大亚湾科技创新研究院）
机构由 AI 辅助整理，请以论文原文为准。
