arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.18546cs.LG
收藏

回归中目标噪声的梯度中的信息

Information Hidden in Gradients of Regression with Target Noise

University of Helsinki(赫尔辛基大学)

Arash Jamshidi, Katsiaryna Haitsiukevich, Kai Puolamäki

AI总结:

研究通过梯度揭示Hessian,利用噪声校准提升优化和鲁棒性,提供非渐近保证。

摘要:

二阶信息如曲率或数据协方差对优化、诊断和鲁棒性至关重要。然而,在许多现代设置中,只有梯度是可观测的。我们展示梯度本身可以揭示Hessian,等于线性回归的数据协方差Σ。我们的关键见解是简单的方差校准:注入高斯噪声使总目标噪声方差等于批量大小,确保经验梯度协方差近似Hessian,即使在远离最优解时。我们提供亚高斯输入下的非渐近算子范数保证。我们还表明,没有此类校准,恢复可能因Ω(1)因子失败。所提方法实用(

英文摘要:

Second-order information -- such as curvature or data covariance -- is critical for optimisation, diagnostics, and robustness. However, in many modern settings, only the gradients are observable. We show that the gradients alone can reveal the Hessian, equalling the data covariance $Σ$ for the linear regression. Our key insight is a simple variance calibration: injecting Gaussian noise so that the total target noise variance equals the batch size ensures that the empirical gradient covariance closely approximates the Hessian, even when evaluated far from the optimum. We provide non-asymptotic operator-norm guarantees under sub-Gaussian inputs. We also show that without such calibration, recovery can fail by an $Ω(1)$ factor. The proposed method is practical (a "set target-noise variance to $n$" rule) and robust (variance $\mathcal{O}(n)$ suffices to recover $Σ$ up to scale). Applications include preconditioning for faster optimisation, adversarial risk estimation, and gradient-only training, for example, in distributed systems. We support our theoretical results with experiments on synthetic and real data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑