Git Re-Basin：合并模置换对称模型
Git Re-Basin: Merging Models modulo Permutation Symmetries
摘要：
深度学习的成功在很大程度上归功于我们相对轻松地解决某些大规模非凸优化问题的能力。虽然非凸优化是NP难的，但简单的算法--通常是随机梯度下降的变体--在实践中拟合大型神经网络时表现出惊人的有效性。我们认为，神经网络损失景观通常包含（几乎）一个单一的盆地后，考虑到所有可能的置换对称性的隐藏单元la Entezari et al. 2021。我们介绍了三种算法来置换一个模型的单元，使它们与参考模型对齐，以便在权重空间中合并两个模型。该变换产生一组功能等效的权重，这些权重位于参考模型附近的近似凸盆中。在实验上，我们在各种模型架构和数据集上展示了单流域现象，包括第一次（据我们所知）在CIFAR-10上独立训练的ResNet模型之间的零障碍线性模式连接。此外，我们还发现了模型宽度和训练时间与模式连接性之间的有趣现象。最后，我们讨论了线性模式连通性假设的缺点，包括一个反例的单盆地理论。
英文摘要：
The success of deep learning is due in large part to our ability to solve certain massive non-convex optimization problems with relative ease. Though non-convex optimization is NP-hard, simple algorithms -- often variants of stochastic gradient descent -- exhibit surprising effectiveness in fitting large neural networks in practice. We argue that neural network loss landscapes often contain (nearly) a single basin after accounting for all possible permutation symmetries of hidden units a la Entezari et al. 2021. We introduce three algorithms to permute the units of one model to bring them into alignment with a reference model in order to merge the two models in weight space. This transformation produces a functionally equivalent set of weights that lie in an approximately convex basin near the reference model. Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained ResNet models on CIFAR-10. Additionally, we identify intriguing phenomena relating model width and training time to mode connectivity. Finally, we discuss shortcomings of the linear mode connectivity hypothesis, including a counterexample to the single basin theory.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
