arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1504.00854cs.AIcs.CLstat.ML
收藏

评价评价蒙特卡罗研究

Evaluation Evaluation a Monte Carlo study

Flinders University of South Australia(南澳大利亚弗林德斯大学)

David M. W. Powers

摘要:

在过去的十年中,人们越来越关注自然语言处理/学习的传统评估方法中所体现的偏见,特别是从信息检索中借用的方法。如果不知道所测试的偶然事件的偏倚和患病率,或者等同于由于偶然性而产生的期望,则简单的条件概率召回率、精确度和准确度作为评价指标(无论是单独还是组合,如F因子)都没有意义。NLP度量中存在的偏见会通过增加系统的偏见来“改进”系统,例如通过使用最常见的值(例如水总是名词)来提高标记和解析分数的做法,而不是试图发现正确的值。科恩Kappa和权力信息的措施进行了讨论,作为无偏见的替代召回和相关的心理上显着的措施DeltaP。在本文中,我们将从理论上分析有偏和无偏的措施,所有这些措施之间的精确关系的特点,以及评估评估措施本身的经验,使用蒙特卡罗模拟。

英文摘要:

Over the last decade there has been increasing concern about the biases embodied in traditional evaluation methods for Natural Language Processing/Learning, particularly methods borrowed from Information Retrieval. Without knowledge of the Bias and Prevalence of the contingency being tested, or equivalently the expectation due to chance, the simple conditional probabilities Recall, Precision and Accuracy are not meaningful as evaluation measures, either individually or in combinations such as F-factor. The existence of bias in NLP measures leads to the 'improvement' of systems by increasing their bias, such as the practice of improving tagging and parsing scores by using most common value (e.g. water is always a Noun) rather than the attempting to discover the correct one. The measures Cohen Kappa and Powers Informedness are discussed as unbiased alternative to Recall and related to the psychologically significant measure DeltaP. In this paper we will analyze both biased and unbiased measures theoretically, characterizing the precise relationship between all these measures as well as evaluating the evaluation measures themselves empirically using a Monte Carlo simulation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑