Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption is that all items in a given test set are equal with regards to difficulty and discriminating power. We propose Item Response Theory (IRT) from psychometrics as an alternative means for gold-standard test-set generation and NLP system evaluation. IRT is able to describe characteristics of individual items - their difficulty and discriminating power - and can account for these characteristics in its estimation of human intelligence or ability for an NLP task. In this paper, we demonstrate IRT by generating a gold-standard test set for Recognizing Textual Entailment. By collecting a large number of human responses and fitting our IRT model, we show that our IRT model compares NLP systems with the performance in a human population and is able to provide more insight into system performance than standard evaluation metrics. We show that a high accuracy score does not always imply a high IRT score, which depends on the item characteristics and the response pattern.
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
Mendoza College of Business, University of Notre Dame(圣母大学门多萨商学院);College of Information and Computer Sciences, UMass Amherst(马萨诸塞大学阿默斯特分校信息与计算机科学学院);Department of Psychology and Human Development, Vanderbilt University(范德堡大学心理学与人类发展系);Department of Computer Science, UMass Lowell(马萨诸塞大学洛厄尔分校计算机科学系);Center for Healthcare Organization & Implementation Research, Bedford VA(贝德福德退伍军人事务部医疗保健组织与实施研究中心)
Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning
Center for Precision Psychiatry, Mass General Hospital Department of Psychiatry, Harvard Medical School(精准精神病学中心,麻省总医院精神病科,哈佛医学院);D-Lab University of California, Berkeley(加州大学伯克利分校D实验室)