arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.03429cs.CL
收藏

这个测试集有多难?利用训练动态进行自然语言推理数据集特征刻画

How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics

National University of Science and Technology POLITEHNICA Bucharest(布加勒斯特理工国立大学);University of Illinois at Chicago(伊利诺伊大学芝加哥分校)

Adrian Cosma, Stefan Ruseti, Mihai Dascalu, Cornelia Caragea

AI总结:

本文提出利用训练动态自动划分NLI测试集难度级别,以减少虚假相关性并构建更具挑战性的评估集,同时证明该方法可用于筛选高效训练子集,提升模型性能评估的真实性。

摘要:

自然语言推理(NLI)评估对于评估语言理解模型至关重要;然而,流行的数据集存在系统性的虚假相关性,人为地抬高了模型的实际性能。为了解决这一问题,我们提出了一种自动创建具有挑战性的测试集的方法,无需依赖人工构建不自然和不真实的示例。我们利用训练动态的方法,将流行NLI数据集的测试集划分为三个难度级别。这种划分显著降低了虚假相关性度量,被标记为最高难度的示例表现出明显下降的性能,并包含更真实和多样化的语言现象。当我们的特征刻画方法应用于训练集时,仅使用一小部分数据训练的模型就能达到与在全量数据集上训练的模型相当的性能,超越了其他数据集特征刻画技术。我们的研究解决了NLI数据集构建中的局限性,为模型性能提供了更真实的评估,并对各种自然语言理解(NLU)应用具有重要意义。

英文摘要:

Natural Language Inference (NLI) evaluation is crucial for assessing language understanding models; however, popular datasets suffer from systematic spurious correlations that artificially inflate actual model performance. To address this, we propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. This categorization significantly reduces spurious correlation measures, with examples labeled as having the highest difficulty showing markedly decreased performance and encompassing more realistic and diverse linguistic phenomena. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset, surpassing other dataset characterization techniques. Our research addresses limitations in NLI dataset construction, providing a more authentic evaluation of model performance with implications for diverse NLU applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑