Recently, Ainsworth et al. showed that using weight matching (WM) to minimize the $L^2$ distance in a permutation search of model parameters effectively identifies permutations that satisfy linear mode connectivity (LMC), where the loss along a linear path between two independently trained models with different seeds remains nearly constant. This paper analyzes LMC using WM, which is useful for understanding stochastic gradient descent's effectiveness and its application in areas like model merging. We first empirically show that permutations found by WM do not significantly reduce the $L^2$ distance between two models, and the occurrence of LMC is not merely due to distance reduction by WM itself. We then demonstrate that permutations can change the directions of the singular vectors, but not the singular values, of the weight matrices in each layer. This finding shows that permutations found by WM primarily align the directions of singular vectors associated with large singular values across models. This alignment brings the singular vectors with large singular values, which determine the model's functionality, closer between the original and merged models, allowing the merged model to retain functionality similar to the original models, thereby satisfying LMC. This paper also analyzes activation matching (AM) in terms of singular vectors and finds that the principle of AM is likely the same as that of WM. Finally, we analyze the difference between WM and the straight-through estimator (STE), a dataset-dependent permutation search method, and show that WM can be more advantageous than STE in achieving LMC among three or more models.
Simultaneous linear connectivity of neural networks modulo permutation
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；McGill University（麦吉尔大学）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）；Google Research（谷歌研究院）；Google DeepMind（谷歌DeepMind）
深度学习的成功在很大程度上归功于我们相对轻松地解决某些大规模非凸优化问题的能力。虽然非凸优化是NP难的，但简单的算法--通常是随机梯度下降的变体--在实践中拟合大型神经网络时表现出惊人的有效性。我们认为，神经网络损失景观通常包含（几乎）一个单一的盆地后，考虑到所有可能的置换对称性的隐藏单元la Entezari et al. 2021。我们介绍了三种算法来置换一个模型的单元，使它们与参考模型对齐，以便在权重空间中合并两个模型。该变换产生一组功能等效的权重，这些权重位于参考模型附近的近似凸盆中。在实验上，我们在各种模型架构和数据集上展示了单流域现象，包括第一次（据我们所知）在CIFAR-10上独立训练的ResNet模型之间的零障碍线性模式连接。此外，我们还发现了模型宽度和训练时间与模式连接性之间的有趣现象。最后，我们讨论了线性模式连通性假设的缺点，包括一个反例的单盆地理论。
Learning Latent Permutations with Gumbel-Sinkhorn Networks
Columbia University（哥伦比亚大学）；Google Brain（谷歌大脑）
排列和匹配是各种潜在变量模型的核心构建块，因为它们允许我们对齐，规范化和排序数据。然而，在这样的模型中学习是困难的，因为这些组合对象的精确边缘化是棘手的。作为回应，本文介绍了一系列新的方法，用于在这种模型中进行端到端学习，这些模型使用连续Sinkhorn算子近似离散最大权重匹配。Sinkhorn迭代是有吸引力的，因为它作为softmax算子的简单，易于实现的模拟。有了这个，我们可以定义Gumbel-Sinkhorn方法，这是Gumbel-Softmax方法（Jang et al. 2016，Maddison 2016 et al. 2016）的扩展，用于潜在匹配的分布。我们证明了我们的方法的有效性，通过在一系列质量不同的任务上超越竞争基线：排序数字，解决拼图游戏，识别蠕虫中的神经信号。
Sparse Equation Matching: A Derivative-Free Learning for General-Order Dynamical Systems
School of Management University of Science and Technology of China（科学技术大学管理学院）；Department of Biostatistics and Bioinformatics Duke University（杜克大学生物统计与生物信息学系）
