arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1802.00323cs.IR
收藏

信息检索中评价指标的相关性与预测

Correlation and Prediction of Evaluation Metrics in Information Retrieval

Mucahid Kutlu, Vivek Khetan, Matthew Lease

摘要:

由于研究人员通常没有时间或空间在任何已发表的研究中提出超过几个评价指标,因此在为新方法设定基线或进行系统性荟萃评价时,很难评估先前方法对未报告指标的相对有效性。虽然共享研究数据将有助于缓解这一问题,但最近鼓励持续共享的尝试基本上没有成功。相反,我们建议通过预测一个或多个报告的指标未报告的指标,使以前的工作在任意指标的相对比较。此外,我们进一步研究预测高成本的评估措施,使用低成本的措施作为一个潜在的战略,以降低评估成本。我们首先使用8个TREC测试集合评估23个IR指标之间的相关性。测量预测误差wrt。R-square和Kendall的tau,我们表明,MAP,P@10和RBP的准确预测可以实现仅使用2-3个其他指标。关于降低评估成本,我们表明,RBP(p=0.95)可以预测高精度使用的措施,只有评估深度为30。总之,我们的研究结果提供了一个有价值的概念验证,我们希望通过其他人提出更复杂的度量预测模型来刺激后续工作。

英文摘要:

Because researchers typically do not have the time or space to present more than a few evaluation metrics in any published study, it can be difficult to assess relative effectiveness of prior methods for unreported metrics when baselining a new method or conducting a systematic meta-review. While sharing of study data would help alleviate this, recent attempts to encourage consistent sharing have been largely unsuccessful. Instead, we propose to enable relative comparisons with prior work across arbitrary metrics by predicting unreported metrics given one or more reported metrics. In addition, we further investigate prediction of high-cost evaluation measures using low-cost measures as a potential strategy for reducing evaluation cost. We begin by assessing the correlation between 23 IR metrics using 8 TREC test collections. Measuring prediction error wrt. R-square and Kendall's tau, we show that accurate prediction of MAP, P@10, and RBP can be achieved using only 2-3 other metrics. With regard to lowering evaluation cost, we show that RBP(p=0.95) can be predicted with high accuracy using measures with only evaluation depth of 30. Taken together, our findings provide a valuable proof-of-concept which we expect to spur follow-on work by others in proposing more sophisticated models for metric prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑