在训练集和测试集之间存在分布偏移的情况下学习鲁棒模型是机器学习中的一个基本挑战。虽然跨环境学习不变特征是一种流行的方法,但它通常假设这些特征在训练集和测试集中均被完全观测到,而这一条件在实践中经常被违背。当模型依赖于测试集中缺失的不变特征时,它们在新环境中的鲁棒性可能会下降。为了解决这个问题,我们引入了一种名为充分不变学习(Sufficient Invariant Learning, SIL)框架的新颖学习原则,该框架侧重于学习不变特征的充分子集,而不是依赖单一特征。在证明了现有不变学习方法的局限性之后,我们提出了一种新算法,即自适应锐度感知群组分布鲁棒优化(Adaptive Sharpness-aware Group Distributionally Robust Optimization, ASGDRO),通过在各个环境中寻找共同的平坦极小值来学习多样化的不变特征。我们从理论上证明,寻找共同的平坦极小值能够基于多样化的不变特征实现鲁棒预测。在包括我们新基准在内的多个数据集上的实证评估,证实了ASGDRO对分布偏移的鲁棒性,并突显了现有方法的局限性。
英文摘要:
Learning robust models under distribution shifts between training and test datasets is a fundamental challenge in machine learning. While learning invariant features across environments is a popular approach, it often assumes that these features are fully observed in both training and test sets, a condition frequently violated in practice. When models rely on invariant features absent in the test set, their robustness in new environments can deteriorate. To tackle this problem, we introduce a novel learning principle called the Sufficient Invariant Learning (SIL) framework, which focuses on learning a sufficient subset of invariant features rather than relying on a single feature. After demonstrating the limitation of existing invariant learning methods, we propose a new algorithm, Adaptive Sharpness-aware Group Distributionally Robust Optimization (ASGDRO), to learn diverse invariant features by seeking common flat minima across the environments. We theoretically demonstrate that finding a common flat minima enables robust predictions based on diverse invariant features. Empirical evaluations on multiple datasets, including our new benchmark, confirm ASGDRO's robustness against distribution shifts, highlighting the limitations of existing methods.