偏微分方程神经网络模型损失景观的粗糙度指数
Roughness Index for Loss Landscapes of Neural Network Models of Partial Differential Equations
AI总结：
本文提出基于随机投影总变差均值和方差的粗糙度指数（RI），用于刻画高维损失景观，并发现深度Galerkin方法在PDE求解中的局部极小值附近景观比深度Ritz方法更平滑。
摘要：
损失景观是刻画和比较神经网络模型的有用工具。分析深度神经网络损失景观的主要挑战在于，这些景观通常在高维空间中高度非凸。本文提出了“粗糙度”概念，用于理解高维空间中的此类景观，并将该技术应用于研究由求解微分方程产生的两类神经网络模型。我们的主要创新在于提出了一个定义良好且易于计算的粗糙度指数（RI），该指数基于在随机采样方向上投影的一维函数的（归一化）总变差的均值和方差。局部极小值处较大的RI暗示了振荡的景观轮廓，并表明一阶优化方法面临严峻挑战。特别地，我们观察到在大多数模型中，沿梯度下降路径的RI呈现先增后减的模式。我们将该方法应用于求解偏微分方程（PDE）时使用的两类损失函数，其中PDE的解由神经网络参数化。我们在这些PDE问题上的实证结果揭示了重要且一致的观察：深度Galerkin方法在其局部极小值附近的景观比深度Ritz方法的景观更不粗糙。
英文摘要：
Loss landscape is a useful tool to characterize and compare neural network models. The main challenge for analysis of loss landscape for the deep neural networks is that they are generally highly non-convex in very high dimensional space. In this paper, we develop "the roughness"concept for understanding such landscapes in high dimensions and apply this technique to study two neural network models arising from solving differential equations. Our main innovation is the proposal of a well-defined and easy-to-compute roughness index (RI) which is based on the mean and variance of the (normalized) total variation for one-dimensional functions projected on randomly sampled directions. A large RI at the local minimizer hints an oscillatory landscape profile and indicates a severe challenge for the first-order optimization method. Particularly, we observe the increasing-then-decreasing pattern for RI along the gradient descent path in most models. We apply our method to two types of loss functions used to solve partial differential equations (PDEs) when the solution of PDE is parametrized by neural networks. Our empirical results on these PDE problems reveal important and consistent observations that the landscapes from the deep Galerkin method around its local minimizers are less rough than the deep Ritz method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
