arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1901.10696cs.IR
收藏

利用得分分布比较统计显著性检验对信息检索的评价

Using Score Distributions to Compare Statistical Significance Tests for Information Retrieval Evaluation

Javier Parapar, David E. Losada, Manuel A. Presedo-Quindimil, Alvaro Barreiro

摘要:

统计显著性检验可以提供证据证明两种方法之间观察到的性能差异不是由于偶然性。在信息检索中,一些研究已经检验了这种测试用于比较搜索系统的有效性和适用性。在这里,我们认为,目前的方法评估统计检验的可靠性遭受一些方法上的弱点,我们提出了一种新的方法来研究检索评价的显着性检验。使用分数分布,我们对多个搜索系统的输出进行建模,从这些模型中生成模拟搜索结果,并使用各种显著性检验进行比较。这种方法的一个关键优势是,我们在完全了解零假设的真实性或虚假性的情况下评估统计检验。这种研究信息检索评价中显著性检验功效的新方法是形式化的、创新的。在这种类型的分析之后,我们发现符号检验和Wilcoxon符号检验都比排列检验和t检验具有更高的功效。符号检验和Wilcoxon符号检验在I类错误方面也具有良好的行为。Bootstrap检验显示出很少的I类错误,但它的功效低于其他测试方法。

英文摘要:

Statistical significance tests can provide evidence that the observed difference in performance between two methods is not due to chance. In Information Retrieval, some studies have examined the validity and suitability of such tests for comparing search systems. We argue here that current methods for assessing the reliability of statistical tests suffer from some methodological weaknesses, and we propose a novel way to study significance tests for retrieval evaluation. Using Score Distributions, we model the output of multiple search systems, produce simulated search results from such models, and compare them using various significance tests. A key strength of this approach is that we assess statistical tests under perfect knowledge about the truth or falseness of the null hypothesis. This new method for studying the power of significance tests in Information Retrieval evaluation is formal and innovative. Following this type of analysis, we found that both the sign test and Wilcoxon signed test have more power than the permutation test and the t-test. The sign test and Wilcoxon signed test also have a good behavior in terms of type I errors. The bootstrap test shows few type I errors, but it has less power than the other methods tested.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑