arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.08062cs.IR
收藏

鼓励信息检索方法创新和多样性的新方法

New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches

Mehmet Deniz Türkmen, Matthew Lease, Mucahid Kutlu

摘要:

在评估活动中,参与者经常探索流行的、最先进的基线的变化,作为实现竞争结果的低风险策略。虽然有效,但这可能导致当地“爬山”,而不是更激进和创新地偏离标准方法。此外,如果许多参与者都建立在类似的基线上,所考虑的办法的总体多样性可能有限。在这项工作中,我们提出了一类新的IR评估指标,旨在促进更大的多样性的方法在评估活动。传统的IR指标侧重于用户体验,而我们的两个“创新”指标则奖励探索更多样化、更高风险的策略,以找到其他系统遗漏的相关文档。对四个TREC集合的实验表明,我们的指标确实通过奖励找到此类罕见相关文档的系统来改变系统排名。这一结果进一步得到了受控合成数据实验和定性分析的支持。此外,我们表明,我们的指标实现了更高的评价稳定性和判别力比我们修改的标准指标。为了支持可重复性,我们共享源代码。

英文摘要:

In evaluation campaigns, participants often explore variations of popular, state-of-the-art baselines as a low-risk strategy to achieve competitive results. While effective, this can lead to local "hill climbing" rather than more radical and innovative departure from standard methods. Moreover, if many participants build on similar baselines, the overall diversity of approaches considered may be limited. In this work, we propose a new class of IR evaluation metrics intended to promote greater diversity of approaches in evaluation campaigns. Whereas traditional IR metrics focus on user experience, our two "innovation" metrics instead reward exploration of more divergent, higher-risk strategies finding relevant documents missed by other systems. Experiments on four TREC collections show that our metrics do change system rankings by rewarding systems that find such rare, relevant documents. This result is further supported by a controlled, synthetic data experiment, and a qualitative analysis. In addition, we show that our metrics achieve higher evaluation stability and discriminative power than the standard metrics we modify. To support reproducibility, we share our source code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑