arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.13606cs.CLcs.LG
收藏

分析数据集中性能不稳定的诅咒:后果,来源和建议

The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Xiang Zhou, Yixin Nie, Hao Tan, Mohit Bansal

摘要:

我们发现,最先进的模型在自然语言推理(NLI)和阅读理解(RC)分析/重音集上的表现可能非常不稳定。这就提出了三个问题:(1)不稳定性将如何影响基于这些分析集得出的结论的可靠性?(2)这种不稳定性从何而来?(3)我们应该如何处理这种不稳定性,有哪些潜在的解决方案?对于第一个问题,我们对分析集进行了深入的实证研究,发现除了不稳定的最终性能外,不稳定性存在于整个训练曲线上。我们还观察到低于预期的分析验证集和标准验证集之间的相关性,质疑目前的模型选择程序的有效性。接下来,为了回答第二个问题,我们对不稳定性的来源给出了理论解释和经验证据,表明不稳定性主要来自分析集内的高样本间相关性。最后,对于第三个问题,我们讨论了减轻不稳定性的初步尝试,并为未来的工作提出了指导方针,例如报告分解的方差,以获得更可解释的结果和模型之间的公平比较。我们的代码可在https://github.com/owenzx/InstabilityAnalysis上公开获取

英文摘要:

We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models. Our code is publicly available at: https://github.com/owenzx/InstabilityAnalysis

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑