近似召回置信区间
Approximate Recall Confidence Intervals
摘要：
召回率是指检索到的相关文档所占的比例，它是信息检索中一个重要的有效性指标，特别是在法律、专利和医学领域。如果文件集太大，无法进行详尽的相关性评估，则可以通过评估文件的随机样本来估计召回率;但也需要表明这种估计的可靠性。在这篇文章中，我们研究了几种估计双尾召回置信区间的方法。我们发现，当前使用的正态近似在许多情况下覆盖率很差，即使进行调整以纠正其不适当的对称性也是如此。基于二项式比率的分析和贝叶斯方法通常更准确，但在小群体上不准确。我们推荐的方法是通过固定的超参数和召回率的后验分布的蒙特卡罗估计，推导出检索和未检索产量的β-二项式后验。我们证明，这种方法提供了平均覆盖率或接近名义水平，在几种情况下，同时是平衡和稳定的。我们提供有关抽样设计的建议，包括将评估分配给检索到的和未检索到的片段，并将拟议的β-二项分布与最近TREC Legal Track迭代的官方报告的正常间隔进行比较。
英文摘要：
Recall, the proportion of relevant documents retrieved, is an important measure of effectiveness in information retrieval, particularly in the legal, patent, and medical domains. Where document sets are too large for exhaustive relevance assessment, recall can be estimated by assessing a random sample of documents; but an indication of the reliability of this estimate is also required. In this article, we examine several methods for estimating two-tailed recall confidence intervals. We find that the normal approximation in current use provides poor coverage in many circumstances, even when adjusted to correct its inappropriate symmetry. Analytic and Bayesian methods based on the ratio of binomials are generally more accurate, but are inaccurate on small populations. The method we recommend derives beta-binomial posteriors on retrieved and unretrieved yield, with fixed hyperparameters, and a Monte Carlo estimate of the posterior distribution of recall. We demonstrate that this method gives mean coverage at or near the nominal level, across several scenarios, while being balanced and stable. We offer advice on sampling design, including the allocation of assessments to the retrieved and unretrieved segments, and compare the proposed beta-binomial with the officially reported normal intervals for recent TREC Legal Track iterations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
