我们真的需要排列吗？模型宽度对线性模式连接的影响
Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivity
AI总结：
本研究发现，通过扩大模型宽度并使用合适的softmax温度校准，无需排列即可实现线性模式连接。
摘要：
最近，Ainsworth等人实证表明，给定两个独立训练的模型，应用一种保持输入输出行为的参数排列，允许两个模型通过低损失线性路径连接。当存在这样的路径时，模型被认为实现了线性模式连接（LMC）。先前的研究，包括Ainsworth等人（2023），报告称实现LMC不仅需要适当的排列搜索，还需要足够宽的模型（例如ResNet-20的32倍宽度乘数）。这被普遍认为是因为增加模型宽度确保了足够的候选排列空间，增加了找到产生LMC的排列的可能性。在本工作中，我们实证表明，即使没有任何排列，仅仅扩大模型也是通过合适的softmax温度校准实现LMC的。我们进一步通过分析中间层输出解释了这种现象出现的原因。具体来说，我们引入了逐层指数加权连接（LEWC），它指出合并模型的每一层输出可以表示为相应原始模型的输出的指数加权和。因此，合并模型的输出与原始模型的集合输出相匹配，促进LMC。据我们所知，这项工作是首次表明扩大模型不仅促进了非线性模式连接，如先前研究所建议，还显著增加了实现线性模式连接的可能性。
英文摘要：
Recently, Ainsworth et al. empirically demonstrated that, given two independently trained models, applying a parameter permutation that preserves the input-output behavior allows the two models to be connected by a low-loss linear path. When such a path exists, the models are said to achieve linear mode connectivity (LMC). Prior studies, including Ainsworth et al.(2023), have reported that achieving LMC requires not only an appropriate permutation search but also sufficiently wide models (e.g., a 32 $\times$ width multiplier for ResNet-20). This is broadly believed to be because increasing the model width ensures a large enough space of candidate permutations, increasing the chance of finding one that yields LMC. In this work, we empirically demonstrate that, even without any permutations, simply widening the models is sufficient for achieving LMC when using a suitable softmax temperature calibration. We further explain why this phenomenon arises by analyzing intermediate layer outputs. Specifically, we introduce layerwise exponentially weighted connectivity (LEWC), which states that the output of each layer of the merged model can be represented as an exponentially weighted sum of the outputs of the corresponding layers of the original models. Consequently the merged model's output matches that of an ensemble of the original models, facilitating LMC. To the best of our knowledge, this work is the first to show that widening the model not only facilitates nonlinear mode connectivity, as suggested in prior research, but also significantly increases the possibility of achieving linear mode connectivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
