Visualizing Loss Functions as Topological Landscape Profiles
Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Arizona State University(亚利桑那州立大学);University of California, Davis(加利福尼亚大学戴维斯分校);Dartmouth College(达特茅斯学院);ICSI(国际计算机科学研究所);UC Berkeley(加利福尼亚大学伯克利分校)
Caleb Geniesse, Jiaqing Chen, Tiankai Xie, Ge Shi, Yaoqing Yang, Dmitriy Morozov, Talita Perciano, Michael W. Mahoney, Ross Maciejewski, Gunther H. Weber
In machine learning, a loss function measures the difference between model predictions and ground-truth (or target) values. For neural network models, visualizing how this loss changes as model parameters are varied can provide insights into the local structure of the so-called loss landscape (e.g., smoothness) as well as global properties of the underlying model (e.g., generalization performance). While various methods for visualizing the loss landscape have been proposed, many approaches limit sampling to just one or two directions, ignoring potentially relevant information in this extremely high-dimensional space. This paper introduces a new representation based on topological data analysis that enables the visualization of higher-dimensional loss landscapes. After describing this new topological landscape profile representation, we show how the shape of loss landscapes can reveal new details about model performance and learning dynamics, highlighting several use cases, including image segmentation (e.g., UNet) and scientific machine learning (e.g., physics-informed neural networks). Through these examples, we provide new insights into how loss landscapes vary across distinct hyperparameter spaces: we find that the topology of the loss landscape is simpler for better-performing models; and we observe greater variation in the shape of loss landscapes near transitions from low to high model performance.
LossLens: Diagnostics for Machine Learning through Loss Landscape Visual Analytics
Arizona State University(亚利桑那州立大学);Dartmouth College(达特茅斯学院);University of California, Davis(加州大学戴维斯分校);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);ICSI(国际计算机科学研究所);University of California at Berkeley(加州大学伯克利分校)
Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models
TMLR Group, Hong Kong Baptist University(香港 Baptist 大学 TMLR 团体);Stanford University(斯坦福大学);Mila - Québec AI Institute(魁北克 AI 院);Université de Montréal(蒙特利尔大学);HEC Montréal(蒙特利尔 HEC 学院);Intel AI Lab(英特尔 AI 实验室)