arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.04550cs.LGstat.ML
收藏

In-N-Out:使用辅助信息进行预训练和自训练,以实现分布外鲁棒性

In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness

Stanford University(斯坦福大学)

Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, Percy Liang

摘要:

考虑一个预测设置,其中包含很少的分布内标记示例和许多分布内和分布外(OOD)的未标记示例。我们的目标是学习一个模型,该模型在分布和OOD方面都表现良好。在这些设置中,辅助信息通常可以便宜地用于每个输入。我们应该如何最好地利用这些辅助信息来完成预测任务?在三个图像和时间序列数据集上,从理论上讲,在多任务线性回归设置中,我们表明:(i)使用辅助信息作为输入特征可以改善分布误差,但可能会损害OOD误差;但是(ii)使用辅助信息作为辅助预训练任务的输出可以改善OOD误差。为了两全其美,我们引入了In-N-Out,它首先用辅助输入训练一个模型,并用它来伪标记所有的分布输入,然后在OOD辅助输出上预训练一个模型,并用伪标记(自训练)微调这个模型。我们从理论和经验上表明,在N-输出优于辅助输入或输出单独在分布和OOD错误。

英文摘要:

Consider a prediction setting with few in-distribution labeled examples and many unlabeled examples both in- and out-of-distribution (OOD). The goal is to learn a model which performs well both in-distribution and OOD. In these settings, auxiliary information is often cheaply available for every input. How should we best leverage this auxiliary information for the prediction task? Empirically across three image and time-series datasets, and theoretically in a multi-task linear regression setting, we show that (i) using auxiliary information as input features improves in-distribution error but can hurt OOD error; but (ii) using auxiliary information as outputs of auxiliary pre-training tasks improves OOD error. To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑