Recent work has established clear links between the generalization performance of trained neural networks and the geometry of their loss landscape near the local minima to which they converge. This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training. To this end, researchers have proposed visualizing the loss landscape through the use of simple dimensionality reduction techniques. However, such visualization methods have been limited by their linear nature and only capture features in one or two dimensions, thus restricting sampling of the loss landscape to lines or planes. Here, we expand and improve upon these in three ways. First, we present a novel "jump and retrain" procedure for sampling relevant portions of the loss landscape. We show that the resulting sampled data holds more meaningful information about the network's ability to generalize. Next, we show that non-linear dimensionality reduction of the jump and retrain trajectories via PHATE, a trajectory and manifold-preserving method, allows us to visualize differences between networks that are generalizing well vs poorly. Finally, we combine PHATE trajectories with a computational homology characterization to quantify trajectory differences.
Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity
Wharton School, University of Pennsylvania(宾夕法尼亚大学沃顿商学院);Peking University(北京大学);Stanford University(斯坦福大学);School of Mathematical Sciences, Peking University(北京大学数学科学学院);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心)
Exploring Landscapes for Better Minima along Valleys
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院过程工程研究所状态关键实验室);University of Chinese Academy of Sciences(中国科学院大学)
Exploring the Landscape of Non-Equilibrium Memories with Neural Cellular Automata
SB Schools, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland(瑞士联邦理工学院(EPFL)SB学院);Department of Computing and Mathematical Sciences, Caltech, Pasadena, CA 91125, USA(加州理工学院计算与数学科学系);Institute for Quantum Information and Matter, Caltech, Pasadena, CA 91125, USA(加州理工学院量子信息与物质研究所);Department of Physics, University of California Berkeley, Berkeley, CA 94720, USA(加州大学伯克利分校物理系)
Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking
TRIUMF;Perimeter Institute for Theoretical Physics(佩里米特理论物理研究所);University of Virginia(弗吉尼亚大学);Computer Science and Biocomplexity Institute(计算机科学与生物复杂性研究所);Department of Physics and Astronomy(物理与天文学系);University of Waterloo(滑铁卢大学)
Exploring Energy Landscapes for Minimal Counterfactual Explanations: Applications in Cybersecurity and Beyond
Netcompany-Intrasoft S.A.(Netcompany-Intrasoft公司);Hellenic Mediterranean University(希腊地中海大学);Harokopio University of Athens(雅典哈罗科皮奥大学);University of Western Macedonia(西马其顿大学)
Exploring Quantum Control Landscape and Solution Space Complexity through Dimensionality Reduction & Optimization Algorithms
School of Computer Science, University College Dublin(计算机科学学院,都柏林大学学院);Centre for Quantum Engineering, Science, and Technology, University College Dublin(量子工程、科学与技术中心,都柏林大学学院);School of Physics, University College Dublin(物理学院,都柏林大学学院)
Learning Dynamic Stability Landscapes in Synchronization Networks
Department of Complexity Science, Postdam Institute for Climate Impact Research, Potsdam, Germany(复杂科学系,波茨坦气候影响研究所,德国波茨坦);Department of Digital Transformation in Energy Systems, Institute of Energy Technology, Technical University of Berlin, Germany(能源系统数字化转型系,能源技术研究所,柏林技术大学,德国);Machine Learning Group, Technical University of Berlin, 10587 Berlin, Germany(机器学习组,柏林技术大学,柏林,德国)
Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy
University of Southern Queensland(南昆士兰大学);Lingnan University(岭南大学);Wuhan University of Technology(武汉理工大学);University of Electronic Science and Technology of China(电子科技大学);Hong Kong Polytechnic University(香港理工大学)
本文系统综述机器遗忘的最新方法、指标、数据集与挑战,并指出其在隐私、安全、公平及可信 AI 中的未来方向。
A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs
Department of Computer Science University of California Los Angeles(计算机科学系,加州大学洛杉矶分校);Bioinformatics IDP University of California Los Angeles(生物信息学IDP,加州大学洛杉矶分校);Department of Computational Medicine University of California Los Angeles(计算医学系,加州大学洛杉矶分校)