Studying neural network loss landscapes provides insights into the nature of the underlying optimization problems. Unfortunately, loss landscapes are notoriously difficult to visualize in a human-comprehensible fashion. One common way to address this problem is to plot linear slices of the landscape, for example from the initial state of the network to the final state after optimization. On the basis of this analysis, prior work has drawn broader conclusions about the difficulty of the optimization problem. In this paper, we put inferences of this kind to the test, systematically evaluating how linear interpolation and final performance vary when altering the data, choice of initialization, and other optimizer and architecture design choices. Further, we use linear interpolation to study the role played by individual layers and substructures of the network. We find that certain layers are more sensitive to the choice of initialization, but that the shape of the linear path is not indicative of the changes in test accuracy of the model. Our results cast doubt on the broader intuition that the presence or absence of barriers when interpolating necessarily relates to the success of optimization.
How does the optimizer implicitly bias the model merging loss landscape?
University of Luxembourg（卢森堡大学）；ETH Zürich, MPI for Intelligent Systems（苏黎世联邦理工学院、智能系统研究所）；Idiap Research Institute（Idiap研究机构）；ELLIS Institute Tübingen, MPI for Intelligent Systems, Tübingen AI Center（图宾根ELLIS研究所、智能系统研究所、图宾根人工智能中心）
On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalization
University of Amsterdam（阿姆斯特丹大学）；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学院）；Amsterdam Machine Learning Lab (AMLab), University of Amsterdam（阿姆斯特丹大学机器学习实验室）
Simultaneous linear connectivity of neural networks modulo permutation
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；McGill University（麦吉尔大学）；Mila Quebec AI Institute（米拉魁北克人工智能研究所）；Google Research（谷歌研究院）；Google DeepMind（谷歌DeepMind）
Benchmarking a Tunable Quantum Neural Network on Trapped-Ion and Superconducting Hardware
Joint Quantum Institute（联合量子研究所）；Department of Physics, University of Maryland, College Park, Maryland 20742, USA（大学物理系，马里兰州学院公园，马里兰州20742，美国）；Applied Research Laboratory for Intelligence and Security（智能与安全应用研究实验室）；National Quantum Laboratory (QLab), University of Maryland, College Park, MD 20742 USA（国家量子实验室（QLab），马里兰州学院公园，MD 20742美国）；Duke Quantum Center（杜克大学量子中心）；Department of Physics, Duke University, Durham, North Carolina 27701, USA（大学物理系，北卡罗来纳州达勒姆，北卡罗来纳州27701，美国）
Raymond and Beverly Sackler School of Physics and Astronomy, Tel Aviv University, Tel Aviv 69978, Israel（雷蒙德与贝弗利·萨克勒物理与天文学学院，特拉维夫大学）；École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland（洛桑联邦理工学院（EPFL），洛桑，瑞士）
