自然语言处理的可复制性分析:多数据集的显著性检验
Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
摘要:
随着来自各种语言、领域和流派的文本数据量的不断增长,在多个数据集上评估NLP算法以确保异构设置之间的一致性能已成为标准。然而,这种多重比较对自然语言处理中的传统统计分析方法提出了重大挑战,并可能导致错误的结论。在本文中,我们提出了一个可复制性分析框架,用于对NLP任务算法之间的多重比较进行统计上的合理分析。我们讨论了这个框架的理论优势,目前,统计不合理的,在NLP文献中的实践,并证明其经验价值在四个应用:多域依存句法分析,多语言词性标注,跨域情感分类和词相似度预测。
英文摘要:
With the ever-growing amounts of textual data from a large variety of languages, domains, and genres, it has become standard to evaluate NLP algorithms on multiple datasets in order to ensure consistent performance across heterogeneous setups. However, such multiple comparisons pose significant challenges to traditional statistical analysis methods in NLP and can lead to erroneous conclusions. In this paper, we propose a Replicability Analysis framework for a statistically sound analysis of multiple comparisons between algorithms for NLP tasks. We discuss the theoretical advantages of this framework over the current, statistically unjustified, practice in the NLP literature, and demonstrate its empirical value across four applications: multi-domain dependency parsing, multilingual POS tagging, cross-domain sentiment classification and word similarity prediction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。