AI 中文总结
本文在重尾噪声下比较裁剪与未裁剪的平均SGD，证明裁剪不总能改善主导精度，并揭示其增加方差或偏移极限点的代价。
AI 中文摘要
梯度裁剪被广泛用于稳定训练，但即使在重尾噪声下，它也不一定能提高平均SGD的统计精度。我们在有限条件$p$阶矩（$p\ge2$）下，推导了裁剪与未裁剪的Polyak-Ruppert平均SGD的有限样本比较。我们的主要结果给出了明确的精度和置信条件，在这些条件下，对于$p>2$，高斯项主导未裁剪的偏差界，因此裁剪不必改善其主导阶。通过平衡裁剪偏差和集中性，我们获得了一个界，其中重尾修正对逆失败概率的依赖是对数而非多项式。在$p=2$时，这改善了主导界的置信依赖性。我们通过精确的一维二次递推证明了未裁剪重尾项的尖锐性，并将比较扩展到投影凸SGD。我们还证明了裁剪的具体代价：每个固定的有限阈值都会增加标量高斯二次型上的渐近方差，而全梯度裁剪在非对称噪声下可以移动极限点。
英文摘要
Gradient clipping is widely used to stabilize training, but it need not improve the statistical accuracy of averaged SGD, even under heavy-tailed noise. We derive a finite-sample comparison of clipped and unclipped Polyak-Ruppert averaged SGD under finite conditional $p$-th moments, $p\ge2$. Our main result gives explicit accuracy and confidence conditions under which, for $p>2$, the Gaussian term dominates the unclipped deviation bound, so clipping need not improve its leading order. By balancing clipping bias and concentration, we obtain a bound in which the heavy-tail correction depends logarithmically rather than polynomially on the inverse failure probability. At $p=2$, this improves the confidence dependence of the leading bound. We establish sharpness of the unclipped heavy-tail term through an exact one-dimensional quadratic recursion and extend the comparison to projected convex SGD. We also prove concrete costs of clipping: every fixed finite threshold increases asymptotic variance on a scalar Gaussian quadratic, while whole-gradient clipping can shift the limiting point under asymmetric noise.
