arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.12136cs.LGcs.AI
收藏

将损失函数可视化为拓扑景观剖面

Visualizing Loss Functions as Topological Landscape Profiles

Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室);Arizona State University(亚利桑那州立大学);University of California, Davis(加利福尼亚大学戴维斯分校);Dartmouth College(达特茅斯学院);ICSI(国际计算机科学研究所);UC Berkeley(加利福尼亚大学伯克利分校)

Caleb Geniesse, Jiaqing Chen, Tiankai Xie, Ge Shi, Yaoqing Yang, Dmitriy Morozov, Talita Perciano, Michael W. Mahoney, Ross Maciejewski, Gunther H. Weber

AI总结:

本文提出基于拓扑数据分析的拓扑景观剖面表示,用于可视化高维损失景观,揭示模型性能与学习动态,并发现性能越好损失景观拓扑越简单、性能转变附近形状变化更大。

摘要:

在机器学习中,损失函数衡量模型预测与真实值(或目标值)之间的差异。对于神经网络模型,可视化损失随模型参数变化的方式,可以揭示所谓损失景观的局部结构(如平滑性)以及底层模型的全局性质(如泛化性能)。虽然已经提出了多种可视化损失景观的方法,但许多方法将采样限制在一个或两个方向上,忽略了这一极高维空间中潜在的相关信息。本文引入一种基于拓扑数据分析的新表示方法,使更高维损失景观的可视化成为可能。在描述了这种新的拓扑景观剖面表示之后,我们展示了损失景观的形状如何揭示模型性能和学习动态的新细节,并重点介绍了若干用例,包括图像分割(如UNet)和科学机器学习(如物理信息神经网络)。通过这些示例,我们为损失景观在不同超参数空间中的变化提供了新的见解:我们发现性能更好的模型其损失景观的拓扑结构更简单;并且我们观察到在模型性能从低到高转变的附近,损失景观形状的变化更大。

英文摘要:

In machine learning, a loss function measures the difference between model predictions and ground-truth (or target) values. For neural network models, visualizing how this loss changes as model parameters are varied can provide insights into the local structure of the so-called loss landscape (e.g., smoothness) as well as global properties of the underlying model (e.g., generalization performance). While various methods for visualizing the loss landscape have been proposed, many approaches limit sampling to just one or two directions, ignoring potentially relevant information in this extremely high-dimensional space. This paper introduces a new representation based on topological data analysis that enables the visualization of higher-dimensional loss landscapes. After describing this new topological landscape profile representation, we show how the shape of loss landscapes can reveal new details about model performance and learning dynamics, highlighting several use cases, including image segmentation (e.g., UNet) and scientific machine learning (e.g., physics-informed neural networks). Through these examples, we provide new insights into how loss landscapes vary across distinct hyperparameter spaces: we find that the topology of the loss landscape is simpler for better-performing models; and we observe greater variation in the shape of loss landscapes near transitions from low to high model performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑