基于最坏情况共享最小化的抗光子限制破坏的鲁棒推广
Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimization
摘要:
鲁棒泛化的目的是解决最具挑战性的数据分布,这些数据分布在训练集中是罕见的,并且包含严重的噪声,即,光子限制的腐败常见的解决方案,如分布鲁棒优化(DRO)专注于最坏情况下的经验风险,以确保在不常见的噪声分布上的低训练误差。然而,由于过度参数化的模型是在稀缺的最坏情况下的数据进行优化,DRO无法产生一个平滑的损失景观,从而努力推广到测试集。因此,我们不是专注于最坏情况下的风险最小化,而是通过惩罚最坏情况分布的锐度来提出SharpDRO,该分布测量学习参数邻居周围的损失变化。通过最坏情况下的锐度最小化,所提出的方法成功地产生一个平坦的损失曲线上的损坏的分布,从而实现鲁棒的推广。此外,通过考虑分布注释是否可用,我们将SharpDRO应用于两个问题设置,并设计了一个最坏情况下的选择过程,以实现鲁棒泛化。理论上,我们证明了SharpDRO具有很好的收敛保证。在实验中,我们使用CIFAR 10/100和ImageNet 30数据集模拟光子限制的腐败,并表明SharpDRO对严重的腐败表现出很强的泛化能力,并超过了众所周知的基线方法,具有很大的性能增益。
英文摘要:
Robust generalization aims to tackle the most challenging data distributions which are rare in the training set and contain severe noises, i.e., photon-limited corruptions. Common solutions such as distributionally robust optimization (DRO) focus on the worst-case empirical risk to ensure low training error on the uncommon noisy distributions. However, due to the over-parameterized model being optimized on scarce worst-case data, DRO fails to produce a smooth loss landscape, thus struggling on generalizing well to the test set. Therefore, instead of focusing on the worst-case risk minimization, we propose SharpDRO by penalizing the sharpness of the worst-case distribution, which measures the loss changes around the neighbor of learning parameters. Through worst-case sharpness minimization, the proposed method successfully produces a flat loss curve on the corrupted distributions, thus achieving robust generalization. Moreover, by considering whether the distribution annotation is available, we apply SharpDRO to two problem settings and design a worst-case selection process for robust generalization. Theoretically, we show that SharpDRO has a great convergence guarantee. Experimentally, we simulate photon-limited corruptions using CIFAR10/100 and ImageNet30 datasets and show that SharpDRO exhibits a strong generalization ability against severe corruptions and exceeds well-known baseline methods with large performance gains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。