ILDAE：评价数据的实例级难度分析
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
摘要：
掌握试题的难易程度可以帮助教师通过精心选择的问题快速评估学生的潜力，通过修改琐碎和困难的问题来提高考试质量。我们能在NLP中提取实例难度的好处吗？为此，我们在23个数据集的大规模设置中进行了实例级难度评估数据分析（ILDAE），并展示了其五个新应用：1）以较少的实例进行高效而准确的评估，节省计算成本和时间，2）通过修复错误和琐碎的实例来提高现有评估数据集的质量，3）基于应用需求选择最佳模型，4）分析数据集特征以指导未来的数据创建，5）可靠地估计域外性能。这些应用程序的综合实验产生了几个有趣的发现，例如使用仅5%实例（通过ILDAE选择）的评估与使用完整数据集的评估的Kendall相关性高达0.93，并且使用难度分数计算加权准确性导致与域外性能的相关性提高了5.2%。我们发布了难度分数，并希望我们的分析和发现将更多地关注这个重要但尚未充分研究的领域，即在评估中利用实例难度。
英文摘要：
Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions. Can we extract such benefits of instance difficulty in NLP? To this end, we conduct Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large-scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient-yet-accurate evaluations with fewer instances saving computational cost and time, 2) improving quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) selecting the best model based on application requirements, 4) analyzing dataset characteristics for guiding future data creation, 5) estimating Out-of-Domain performance reliably. Comprehensive experiments for these applications result in several interesting findings, such as evaluation using just 5% instances (selected via ILDAE) achieves as high as 0.93 Kendall correlation with evaluation using complete dataset and computing weighted accuracy using difficulty scores leads to 5.2% higher correlation with Out-of-Domain performance. We release the difficulty scores and hope our analyses and findings will bring more attention to this important yet understudied field of leveraging instance difficulty in evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
