通过测试集难度测试了解深度学习性能:心理测量案例研究
Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study
摘要:
解释深度学习模型的性能超出测试集精度是具有挑战性的。在评估过程中,通常不考虑单个数据点的特征,对每个数据点都一视同仁。我们研究了测试集问题的难度的影响,以确定是否有难度和性能之间的关系。我们使用经过充分研究的心理测量方法对人类的反应模式进行建模。自然语言推理(NLI)和情感分析(SA)的实验表明,正确回答问题的可能性受到问题难度的影响。随着DNN使用更多数据进行训练,简单的示例比困难的示例学习得更快。
英文摘要:
Interpreting the performance of deep learning models beyond test set accuracy is challenging. Characteristics of individual data points are often not considered during evaluation, and each data point is treated equally. We examine the impact of a test set question's difficulty to determine if there is a relationship between difficulty and performance. We model difficulty using well-studied psychometric methods on human response patterns. Experiments on Natural Language Inference (NLI) and Sentiment Analysis (SA) show that the likelihood of answering a question correctly is impacted by the question's difficulty. As DNNs are trained with more data, easy examples are learned more quickly than hard examples.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。