arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1605.08889cs.CL
收藏

运用项目反应理论编制评价量表

Building an Evaluation Scale using Item Response Theory

University of Massachusetts(马萨诸塞大学);Boston College(波士顿学院);Bedford VAMC and CHOIR(贝德福德退伍军人医疗中心及CHOIR)

John P. Lalor, Hao Wu, Hong Yu

摘要:

NLP方法的评估需要对先前审查过的黄金标准测试集进行测试,并报告标准指标(准确性/精确度/召回率/F1)。目前的假设是,在一个给定的测试集的所有项目都是平等的难度和辨别力。我们提出了项目反应理论(IRT)从心理测量学作为一种替代手段的黄金标准测试集生成和NLP系统评估。IRT能够描述单个项目的特征-它们的难度和辨别力-并且可以在其对人类智力或NLP任务能力的估计中解释这些特征。在本文中,我们通过生成一个用于识别文本蕴涵的黄金标准测试集来演示IRT。通过收集大量的人类响应和拟合我们的IRT模型,我们表明,我们的IRT模型比较了NLP系统与人类群体的性能,并且能够比标准评估指标更深入地了解系统性能。我们发现,一个高的准确性分数并不总是意味着一个高的IRT分数,这取决于项目的特点和反应模式。

英文摘要:

Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption is that all items in a given test set are equal with regards to difficulty and discriminating power. We propose Item Response Theory (IRT) from psychometrics as an alternative means for gold-standard test-set generation and NLP system evaluation. IRT is able to describe characteristics of individual items - their difficulty and discriminating power - and can account for these characteristics in its estimation of human intelligence or ability for an NLP task. In this paper, we demonstrate IRT by generating a gold-standard test set for Recognizing Textual Entailment. By collecting a large number of human responses and fitting our IRT model, we show that our IRT model compares NLP systems with the performance in a human population and is able to provide more insight into system performance than standard evaluation metrics. We show that a high accuracy score does not always imply a high IRT score, which depends on the item characteristics and the response pattern.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑