多元中的和谐：利用典型相关分析合并神经网络
Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis
AI总结：
针对多模型集成成本高且模型融合受限于神经元一一排列的问题，提出基于典型相关分析的 CCA Merge，通过对齐特征线性组合提升两模型及多模型合并性能。
摘要：
通过集成结合多个已训练模型的预测，通常是利用模型所学到的不同特征来提高准确率的好方法，但这会带来高昂的计算和存储成本。模型融合是通过组合多个模型的参数将它们合并为一个模型的操作，可以降低这些成本，但在实践中效果并不理想。事实上，神经网络的损失景观是高维且非凸的，通过学习找到的极小值之间通常被高损失势垒隔开。近期大量工作一直专注于寻找将一个网络的特征与另一个网络的特征相匹配的排列，以降低参数空间中二者之间线性路径上的损失势垒。然而，排列具有局限性，因为它们假设不同模型的神经元之间存在一一映射。我们提出了一种新的模型合并算法 CCA Merge，该算法基于典型相关分析（Canonical Correlation Analysis），旨在最大化模型特征线性组合之间的相关性。我们表明，在对在相同或不同数据划分上训练的模型进行平均时，我们的对齐方法比以往方法带来了更好的性能。我们还将这一分析扩展到合并超过2个模型这一更困难的设置，并发现 CCA Merge 的效果显著优于以往方法。我们的代码可在 https://github.com/shoroi/align-n-merge 公开获取。
英文摘要：
Combining the predictions of multiple trained models through ensembling is generally a good way to improve accuracy by leveraging the different learned features of the models, however it comes with high computational and storage costs. Model fusion, the act of merging multiple models into one by combining their parameters reduces these costs but doesn't work as well in practice. Indeed, neural network loss landscapes are high-dimensional and non-convex and the minima found through learning are typically separated by high loss barriers. Numerous recent works have been focused on finding permutations matching one network features to the features of a second one, lowering the loss barrier on the linear path between them in parameter space. However, permutations are restrictive since they assume a one-to-one mapping between the different models' neurons exists. We propose a new model merging algorithm, CCA Merge, which is based on Canonical Correlation Analysis and aims to maximize the correlations between linear combinations of the model features. We show that our alignment method leads to better performances than past methods when averaging models trained on the same, or differing data splits. We also extend this analysis into the harder setting where more than 2 models are merged, and we find that CCA Merge works significantly better than past methods. Our code is publicly available at https://github.com/shoroi/align-n-merge
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
