arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.09044cs.LGcs.AIcs.CYstat.ML
收藏

只需训练两次:无需训练组信息即可提高组鲁棒性

Just Train Twice: Improving Group Robustness without Training Group Information

Stanford University(斯坦福大学)

Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, Chelsea Finn

摘要:

通过经验风险最小化(ERM)的标准训练可以产生平均准确度高但对某些群体准确度低的模型,特别是在输入和标签之间存在虚假相关性的情况下。实现高最差组准确度的先前方法,如组分布鲁棒优化(组DRO),需要针对每个训练点的昂贵的组注释,而不使用这种组注释的方法通常实现不令人满意的最差组准确度。在本文中,我们提出了一个简单的两阶段的方法,JTT,第一次训练一个标准的ERM模型几个时期,然后训练第二个模型,增加权重的训练样本,第一个模型错误分类。直观地说,这增加了标准ERM模型表现不佳的组的示例,从而改善了最差组的表现。平均超过四个图像分类和自然语言处理任务与虚假的相关性,JTT关闭75%的差距,在最坏的组之间的准确性标准ERM和组DRO,而只需要组注释的一个小的验证集,以调整超参数。

英文摘要:

Standard training via empirical risk minimization (ERM) can produce models that achieve high accuracy on average but low accuracy on certain groups, especially in the presence of spurious correlations between the input and label. Prior approaches that achieve high worst-group accuracy, like group distributionally robust optimization (group DRO) require expensive group annotations for each training point, whereas approaches that do not use such group annotations typically achieve unsatisfactory worst-group accuracy. In this paper, we propose a simple two-stage approach, JTT, that first trains a standard ERM model for several epochs, and then trains a second model that upweights the training examples that the first model misclassified. Intuitively, this upweights examples from groups on which standard ERM models perform poorly, leading to improved worst-group performance. Averaged over four image classification and natural language processing tasks with spurious correlations, JTT closes 75% of the gap in worst-group accuracy between standard ERM and group DRO, while only requiring group annotations on a small validation set in order to tune hyperparameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑