Benchmark中的所有数据集都是必要的吗?文本分类中数据集评价的初步研究
Are All the Datasets in Benchmark Necessary? A Pilot Study of Dataset Evaluation for Text Classification
摘要:
在本文中,我们提出的研究问题,是否所有的数据集在基准是必要的。我们首先通过比较不同系统时表征数据集的可重复性来实现这一点。在9个数据集和36个系统上的实验表明,几个现有的基准数据集对区分得分最高的系统贡献不大,而那些较少使用的数据集表现出令人印象深刻的区分能力。我们进一步以文本分类任务为案例研究,研究了基于其属性(例如,平均句子长度)。我们的初步实验令人鼓舞地表明,给定足够数量的训练实验记录,可以学习一个有意义的预测器来估计对未知数据集的数据集区分。我们在DataLab:\url{https://datalab.nlpedia.ai}上发布了本工作中探索的所有数据集。
英文摘要:
In this paper, we ask the research question of whether all the datasets in the benchmark are necessary. We approach this by first characterizing the distinguishability of datasets when comparing different systems. Experiments on 9 datasets and 36 systems show that several existing benchmark datasets contribute little to discriminating top-scoring systems, while those less used datasets exhibit impressive discriminative power. We further, taking the text classification task as a case study, investigate the possibility of predicting dataset discrimination based on its properties (e.g., average sentence length). Our preliminary experiments promisingly show that given a sufficient number of training experimental records, a meaningful predictor can be learned to estimate dataset discrimination over unseen datasets. We released all datasets with features explored in this work on DataLab: \url{https://datalab.nlpedia.ai}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。