arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05417cs.IR
收藏

纵向信息检索评估的可复现性度量

Replicability Measures for Longitudinal Information Retrieval Evaluation

Jüri Keller, Timo Breuer, Philipp Schaer

AI总结:

本研究基于 LongEval 共享任务,将 IR 系统有效性随时间的持续性视为可复现性问题,采用调整后的可复现性度量分析系统性能衰减与排名变化,发现最有效系统未必性能最持久。

摘要:

信息检索(IR)系统的大多数组成部分都处于不断变化之中。文档会被创建、更新或删除,信息需求也在变化,甚至相关性可能也不是静态的。尽管人们普遍期望 IR 系统能为用户保持一致的效用,但测试集评估依赖于固定的实验设置。基于 LongEval 共享任务和测试集,本工作探索了如何评估在不断演变的实验中测得的有效性。具体而言,有效性的持续性被作为一项可复现性任务进行研究。研究观察到,与初始测量相比,有效性如何随时间逐渐下降。采用经过调整的可复现性度量,可以进一步洞察有效性的持续性。系统的排名会随检索度量和时间的不同而变化。总之,研究发现,最有效的系统并不一定是性能最持久的系统。

英文摘要:

Information Retrieval (IR) systems are exposed to constant changes in most components. Documents are created, updated, or deleted, the information needs are changing, and even relevance might not be static. While it is generally expected that the IR systems retain a consistent utility for the users, test collection evaluations rely on a fixed experimental setup. Based on the LongEval shared task and test collection, this work explores how the effectiveness measured in evolving experiments can be assessed. Specifically, the persistency of effectiveness is investigated as a replicability task. It is observed how the effectiveness progressively deteriorates over time compared to the initial measurement. Employing adapted replicability measures provides further insight into the persistence of effectiveness. The ranking of systems varies across retrieval measures and time. In conclusion, it was found that the most effective systems are not necessarily the ones with the most persistent performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑