我们需要谈谈随机分裂
We Need to Talk About Random Splits
摘要：
Gorman和Bedrick（2019）主张在NLP实验中使用随机分裂而不是标准分裂。我们认为，随机分割，像标准分割，导致过于乐观的性能估计。我们还可以以有偏见或敌对的方式分割数据，例如，短句训练和长句评价。有偏采样已用于领域自适应，以模拟真实世界的漂移;这被称为协变量偏移假设。然而，在NLP中，即使是最坏情况的分裂，最大化偏差，也经常低估在域内数据的新样本上观察到的误差，即，模型在测试时最低限度地泛化到的数据。这使协变量偏移假设无效。而不是使用多个随机分裂，未来的基准应该包括多个，独立的测试集，而不是不可行的，我们认为，多个有偏见的分裂导致更现实的性能估计比多个随机分裂。
英文摘要：
Gorman and Bedrick (2019) argued for using random splits rather than standard splits in NLP experiments. We argue that random splits, like standard splits, lead to overly optimistic performance estimates. We can also split data in biased or adversarial ways, e.g., training on short sentences and evaluating on long ones. Biased sampling has been used in domain adaptation to simulate real-world drift; this is known as the covariate shift assumption. In NLP, however, even worst-case splits, maximizing bias, often under-estimate the error observed on new samples of in-domain data, i.e., the data that models should minimally generalize to at test time. This invalidates the covariate shift assumption. Instead of using multiple random splits, future benchmarks should ideally include multiple, independent test sets instead; if infeasible, we argue that multiple biased splits leads to more realistic performance estimates than multiple random splits.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
