PGrad:面向域泛化的主梯度学习
PGrad: Learning Principal Gradients For Domain Generalization
AI总结:
针对域泛化问题,提出PGrad方法,通过聚合训练轨迹主方向学习鲁棒梯度,结合双射优化与方向长度校准,经理论证明和多基准实验验证,在分布偏移场景下取得有竞争力的泛化效果。
摘要:
机器学习模型在面对分布外(OOD)域时性能会下降,这一挑战性任务被称为域泛化(DG)。本研究提出了一种名为PGrad的新型DG训练策略,用于学习鲁棒的梯度方向,提升模型在未见过的域上的泛化能力。该方法的梯度聚合了采样的滚动优化轨迹的主方向,该轨迹用于衡量所有训练域上的训练动态。PGrad的梯度设计迫使DG训练忽略依赖域的噪声信号,并以覆盖参数动态主要成分的鲁棒方向更新所有训练域。我们进一步通过基于双射的计算优化以及基于方向和长度的校准改进了PGrad。我们的理论证明将PGrad与神经网络训练中的Hessian谱分析联系起来。在DomainBed和WILDS基准上的实验表明,我们的方法有效实现了鲁棒的DG优化,并带来了平滑下降的损失曲线。实证结果显示,PGrad在七个数据集上取得了具有竞争力的结果,证明了其在合成和真实分布偏移下的有效性。代码可在https://github.com/QData/PGrad获取。
英文摘要:
Machine learning models fail to perform when facing out-of-distribution (OOD) domains, a challenging task known as domain generalization (DG). In this work, we develop a novel DG training strategy, we call PGrad, to learn a robust gradient direction, improving models' generalization ability on unseen domains. The proposed gradient aggregates the principal directions of a sampled roll-out optimization trajectory that measures the training dynamics across all training domains. PGrad's gradient design forces the DG training to ignore domain-dependent noise signals and updates all training domains with a robust direction covering main components of parameter dynamics. We further improve PGrad via bijection-based computational refinement and directional plus length-based calibrations. Our theoretical proof connects PGrad to the spectral analysis of Hessian in training neural networks. Experiments on DomainBed and WILDS benchmarks demonstrate that our approach effectively enables robust DG optimization and leads to smoothly decreased loss curves. Empirically, PGrad achieves competitive results across seven datasets, demonstrating its efficacy across both synthetic and real-world distributional shifts. Code is available at https://github.com/QData/PGrad.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。