RanSOM：具有随机缩放的二次动量用于约束和无约束优化
RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
AI总结：
RanSOM通过引入随机步长消除动量方法中的曲率偏差，实现无约束和约束优化的最优收敛率。
摘要：
RanSOM通过引入随机步长消除动量方法中的曲率偏差，实现无约束和约束优化的最优收敛率。
英文摘要：
Second-order momentum corrects the lag of momentum behind the current gradient with a Hessian-vector product (HVP), but existing corrections need either a Lipschitz Hessian or extra forward and backward passes at an auxiliary point. We show that randomizing the step length removes this trade-off: if the step is exponentially distributed with mean $η$, a Stein-type identity makes $η$ times one HVP at the new iterate an estimate of the gradient change that is unbiased in expectation over the step, and this HVP shares its forward pass with the next gradient. The resulting method, RanSOM-E, works with normalized, sign, and Muon-type steps, and a Beta-distributed variant, RanSOM-B, keeps Frank-Wolfe iterates feasible. For objectives with Lipschitz gradient and heavy-tailed gradient and Hessian noise with tail indices $p, q \in (1,2]$, RanSOM-E attains the optimal oracle-complexity exponents for every pair $(p,q)$, including $p \neq q$, without a Lipschitz Hessian or clipping ($\mathcal{O}(\varepsilon^{-3})$ for $p = q = 2$). On CIFAR-10 with ResNet-20, we detect no accuracy difference from existing second-order momentum methods, and its mean update time is lower than that of the auxiliary-point method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
