使用大学习率探索平坦极小值以实现域泛化
Exploring Flat Minima for Domain Generalization with Large Learning Rates
AI总结:
本文提出利用大学习率促进权重多样性以寻找平坦极小值,并引入Lookahead策略通过快慢权重插值解决收敛问题,在域泛化分类和分割基准上取得SOTA性能。
摘要:
域泛化(DG)旨在泛化到任意未见过的域。在DG中改进模型泛化能力的一种有前景的方法是寻找平坦极小值。此任务的一个典型方法是SWAD,它涉及沿训练轨迹平均权重。然而,权重平均的成功取决于权重的多样性,而在使用小学习率训练时,这种多样性是有限的。相反,我们观察到利用大学习率可以同时促进权重多样性并有助于在损失景观中寻找平坦区域。但是,采用大学习率会面临收敛问题,这无法通过简单地平均训练权重来解决。为了解决这个问题,我们引入了一种名为Lookahead的训练策略,它涉及快权重和慢权重之间的权重插值,而不是平均。快权重使用大学习率探索权重空间,此时并未收敛,而慢权重通过与其插值来确保收敛。此外,权重插值还通过隐式优化衡量平坦度的局部熵损失来帮助寻找平坦极小值。为了在训练期间进一步防止过拟合,我们提出了两种变体,使用加权平均权重或累积历史权重来正则化训练权重。利用这一新视角,我们的方法在分类和语义分割域泛化基准上均取得了最先进的性能。代码可在 https://github.com/koncle/DG-with-Large-LR 获取。
英文摘要:
Domain Generalization (DG) aims to generalize to arbitrary unseen domains. A promising approach to improve model generalization in DG is the identification of flat minima. One typical method for this task is SWAD, which involves averaging weights along the training trajectory. However, the success of weight averaging depends on the diversity of weights, which is limited when training with a small learning rate. Instead, we observe that leveraging a large learning rate can simultaneously promote weight diversity and facilitate the identification of flat regions in the loss landscape. However, employing a large learning rate suffers from the convergence problem, which cannot be resolved by simply averaging the training weights. To address this issue, we introduce a training strategy called Lookahead which involves the weight interpolation, instead of average, between fast and slow weights. The fast weight explores the weight space with a large learning rate, which is not converged while the slow weight interpolates with it to ensure the convergence. Besides, weight interpolation also helps identify flat minima by implicitly optimizing the local entropy loss that measures flatness. To further prevent overfitting during training, we propose two variants to regularize the training weight with weighted averaged weight or with accumulated history weight. Taking advantage of this new perspective, our methods achieve state-of-the-art performance on both classification and semantic segmentation domain generalization benchmarks. The code is available at https://github.com/koncle/DG-with-Large-LR.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。