发表机构
Texas State University（德克萨斯州立大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究定义了判断Adam停滞的损失景观相关指标，通过$2\times2$示例和FINER图像拟合案例说明Adam的对角预条件无法消除交叉耦合病态性，为分析Adam的收敛特性提供了基于预条件海森的视角。
AI 中文摘要
在隐式神经表示（INR）架构和分析基准中，我们观察到经过充分调优的Adam（尤其是其学习率（lr），例如在从$lr=0.05$到$10^{-8}$的超参数搜索中）即使在病态损失景观上也能达到极低的损失，或在远高于二阶方法所得损失的平台处收敛。本报告定义了用于确定Adam能否缓解给定损失景观病态性的测量指标。我们提供了确定每种结果的指标，包括海森矩阵的条件数、经Adam预条件的海森矩阵$D^{-1/2}HD^{-1/2}$的条件数（附Adam更新规则的推导）、区分轴对齐与交叉耦合病态性的对角质量$\rho$、通过随机兰索斯求积估计的负谱质量，以及曲率带上的梯度能量分数，其中包含指示Adam停滞的平坦分数。一个详细的$2\times2$示例和图示说明了Adam的对角预条件为何能通过重新缩放消除轴对齐病态性，而当病态性为交叉耦合时则无法做到。此外，我们还呈现了FINER图像拟合架构的案例研究，涵盖了完整的损失景观分析框架：拟合架构描述、其景观在鞍点处使Adam停滞的原因、我们调优的基线至分块二阶方法的$120$--$134$\text{dB}结果的测得PSNR值、这些数字背后的误差图，以及此类图像拟合精度在实践中带来的益处的描述。
英文摘要
Across implicit-neural-representation (INR) architectures and analytic benchmarks we observe that a thoroughly tuned Adam (especially its learning rate (lr), e.g. in a hyperparameter sweep from $lr = 0.05$ to $10^{-8}$) can potentially reach a very low loss even on ill-conditioned loss landscape or converge at a plateau far above the loss attained by second-order methods. This report defines the measured metrics that help determine if Adam can mitigate the ill-conditioning on a given loss landscape. We provide the indicators by which each outcome is determined, that are: the condition number of the Hessian and of the Adam-preconditioned Hessian $D^{-1/2}HD^{-1/2}$ (with the derivation from Adam's update rule), the diagonal mass $ρ$ that distinguishes axis-aligned from cross-coupled ill-conditioning, the negative spectral mass estimated by stochastic Lanczos quadrature, and the gradient energy fractions over curvature bands, including the flat fraction that indicates the Adam stall. A worked out $2\times 2$ example and an illustration show the reasons why a diagonal preconditioning by Adam can remove axis-aligned ill-conditioning by rescaling and why it cannot do the same if the ill-conditioning is cross coupled. In addition, we present a case study of FINER image fitting architecture that goes over the whole loss landscape analysis framework: the fitting architecture description, reasons due to which its landscape stalls Adam at saddles, the measured PSNR values through our tuned baselines to the $120$--$134$\,dB results of the blockwise second order methods, the error maps behind those numbers, and description of the benefits such image fitting accuracy gives in practice.
