arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2506.04487cs.LGstat.ML
收藏

OrthoGrad 改善神经校准

OrthoGrad Improves Neural Calibration

University of Denver(丹佛大学)

C. Evans Hedges

AI总结:

本研究提出几何感知优化方法 OrthoGrad,通过正交化梯度更新约束下降方向,在不改架构下降低过度自信,提升校准与不确定性估计,且开销极小。

摘要:

我们研究 $\perp$Grad,一种对基于梯度的优化进行几何感知修改的方法,该方法约束下降方向以解决过度自信问题,这是标准优化器在不确定性关键应用中的一个关键局限。通过强制梯度更新与权重向量之间的正交性,$\perp$Grad 在不改变架构的情况下改变优化轨迹。在 CIFAR-10 上使用 10% 标记数据时,$\perp$Grad 在准确率上与 SGD 相当,同时在测试损失($p=0.05$)、预测熵($p=0.001$)和置信度度量方面取得了统计显著的改进。这些效应在腐败水平和架构上表现出一致的趋势。$\perp$Grad 与优化器无关,开销极小,并且与事后校准技术兼容。理论上,我们刻画了简化 $\perp$Grad 变体的收敛性和驻点,揭示正交化约束损失减少路径以避免置信度膨胀并促进决策边界改进。我们的研究结果表明,优化中的几何干预可以在低计算成本下改善预测不确定性估计。

英文摘要:

We study $\perp$Grad, a geometry-aware modification to gradient-based optimization that constrains descent directions to address overconfidence, a key limitation of standard optimizers in uncertainty-critical applications. By enforcing orthogonality between gradient updates and weight vectors, $\perp$Grad alters optimization trajectories without architectural changes. On CIFAR-10 with 10% labeled data, $\perp$Grad matches SGD in accuracy while achieving statistically significant improvements in test loss ($p=0.05$), predictive entropy ($p=0.001$), and confidence measures. These effects show consistent trends across corruption levels and architectures. $\perp$Grad is optimizer-agnostic, incurs minimal overhead, and remains compatible with post-hoc calibration techniques. Theoretically, we characterize convergence and stationary points for a simplified $\perp$Grad variant, revealing that orthogonalization constrains loss reduction pathways to avoid confidence inflation and encourage decision-boundary improvements. Our findings suggest that geometric interventions in optimization can improve predictive uncertainty estimates at low computational cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑