发表机构
University of Massachusetts Amherst; Flatiron Institute; Cornell University（马萨诸塞大学阿默斯特分校; 熨斗研究所; 康奈尔大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
SoftServe提出一族无需线搜索的拟牛顿方法，通过正定曲率估计和Newton-Schulz迭代实现大规模深度学习优化，在病态任务上优于Adam等基线。
AI 中文摘要
拟牛顿（QN）方法长期以来一直是大规模无约束凸优化中最有效的方法之一。有两个障碍限制了它们在深度学习中的应用：非凸性和巨大的参数规模。我们引入了SoftServe，这是一族旨在克服这些障碍的QN方法，无需线搜索或临时的曲率修正。SoftServe从Berglund等人（2025）的变分目标中推导出正定曲率估计，即使在存在负曲率的情况下也是如此。我们开发了对角线和Kronecker因子化的变体，这些变体通过构造保持正定性，并可扩展到巨大的神经网络。最后，SoftServe依赖稳定的耦合Newton-Schulz迭代进行所需的矩阵运算，用适合GPU的矩阵乘法替代了昂贵的矩阵分解。SoftServe在严重病态的问题上表现出色，包括循环网络、深度自编码器、物理信息神经网络以及一个1.36亿参数的物理信息扩散模型等任务，通常能达到比包括Adam、Muon和SOAP在内的既有基线更低的损失。
英文摘要
Quasi-Newton (QN) methods have long been among the most effective methods for large-scale unconstrained convex optimization. Two obstacles have limited their use in deep learning: non-convexity and enormous parameter sizes. We introduce SoftServe, a family of QN methods designed to overcome these obstacles without line searches or ad hoc curvature corrections. SoftServe derives positivedefinite curvature estimates from the variational objective of Berglund et al. (2025), even in the presence of negative curvature. We develop diagonal and Kroneckerfactored variants that preserve positive definiteness by construction and scale to massive neural networks. Finally, SoftServe relies on the stable coupled Newton-Schulz iteration for the required matrix operations, replacing costly matrix decompositions with GPU-friendly matrix multiplications. SoftServe excels on problems that are severely ill-conditioned, including tasks such as recurrent networks, deep autoencoders, physics-informed neural networks, and a 136M-parameter physics-informed diffusion model, often achieving lower losses than established baselines including Adam, Muon, and SOAP.
