M-Loss：利用有限无标签数据量化模型合并兼容性
M-Loss: Quantifying Model Merging Compatibility with Limited Unlabeled Data
AI总结：
M-Loss通过有限无标签数据量化模型合并兼容性，提供理论可行性和参数重要性的指导，提升模型整合效率。
摘要：
大规模模型的训练既计算密集又常受限于标记数据的可用性。模型合并提供了一个有吸引力的替代方案，通过直接整合多个源模型的权重而不需额外数据或大量训练。然而，传统模型合并技术，如参数平均，往往导致不可泛化的特征的意外组合，尤其是在源模型权重差异显著时。相比之下，模型融合通常能提供更稳定和优越的性能，通过平均输出聚合多个模型。但其会带来更高的推理成本和存储需求。尽管先前研究已实验性展示了模型合并与融合之间的相似性，但理论证据和评估指标仍显不足。为此，我们引入了合并-融合损失（M-Loss），一种新的评估指标，通过非常有限的无标签数据量化合并源模型的兼容性。通过在层和节点级别测量参数平均与模型融合之间的差异，M-Loss促进了更有效的合并策略。具体而言，M-Loss既作为模型合并理论可行性的量化标准，也作为模型剪枝中参数重要性的指导。我们的理论分析和实证评估表明，将M-Loss纳入合并过程显著提高了合并模型与模型融合之间的对齐程度，提供了一个可扩展且高效的准确模型整合框架。
英文摘要：
Training of large-scale models is both computationally intensive and often constrained by the availability of labeled data. Model merging offers a compelling alternative by directly integrating the weights of multiple source models without requiring additional data or extensive training. However, conventional model merging techniques, such as parameter averaging, often suffer from the unintended combination of non-generalizable features, especially when source models exhibit significant weight disparities. Comparatively, model ensembling generally provides more stable and superior performance that aggregates multiple models by averaging outputs. However, it incurs higher inference costs and increased storage requirements. While previous studies experimentally showed the similarities between model merging and ensembling, theoretical evidence and evaluation metrics remain lacking. To address this gap, we introduce Merging-ensembling loss (M-Loss), a novel evaluation metric that quantifies the compatibility of merging source models using very limited unlabeled data. By measuring the discrepancy between parameter averaging and model ensembling at layer and node levels, M-Loss facilitates more effective merging strategies. Specifically, M-Loss serves both as a quantitative criterion of the theoretical feasibility of model merging, and a guide for parameter significance in model pruning. Our theoretical analysis and empirical evaluations demonstrate that incorporating M-Loss into the merging process significantly improves the alignment between merged models and model ensembling, providing a scalable and efficient framework for accurate model consolidation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
