不依赖人类反应模式的潜在参数学习:人工群体的项目反应理论
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
摘要:
将项目反应理论(IRT)应用到NLP任务中可以提供关于模型性能和行为的有价值的信息。传统上,IRT模型是使用人类响应模式(RP)数据来学习的,这对于训练深度神经网络(DNN)所需的大型数据集来说是一个重大瓶颈。在这项工作中,我们提出了使用从DNN模型的人工人群中生成的RP来学习IRT模型。我们通过对两个NLP任务的定量和定性分析,证明了使用DNN生成的数据学习IRT模型的有效性。从人类和机器RP学习的用于自然语言推理和情感分析的参数表现出中等到大的正相关性。我们展示了一个潜在难度项目参数的用例,即训练集过滤,并表明使用难度对训练数据进行采样优于基线方法。最后,我们强调的情况下,人类对项目难度的期望不匹配的机器RP估计的难度。
英文摘要:
Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。