arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.08663cs.IRcs.AIcs.CL
收藏

BEIR:一个用于信息检索模型Zero-shot评估的异质基准

BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models

Technische Universität Darmstadt(达姆施塔特工业大学)

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych

摘要:

现有的神经信息检索(IR)模型通常是在同质和狭窄的环境中进行研究的,这大大限制了对它们的分布外(OOD)泛化能力的了解。为了解决这个问题,并方便研究人员广泛地评估他们的模型的有效性,我们引入基准IR(BEIR),一个强大的和异构的评估基准信息检索。我们从不同的文本检索任务和领域中仔细选择了18个公开可用的数据集,并在BEIR基准上评估了10个最先进的检索系统,包括词法,稀疏,密集,后期交互和重新排名架构。我们的研究结果表明,BM 25是一个强大的基线和重新排名和后期互动为基础的模型平均达到最佳的zero-shot性能,但是,在高计算成本。相比之下,密集和稀疏检索模型的计算效率更高,但往往表现不佳的其他方法,突出了相当大的改进空间,在其泛化能力。我们希望这个框架可以让我们更好地评估和理解现有的检索系统,并有助于加快在未来更好的鲁棒性和可推广的系统的进展。BEIR可在https://github.com/UKPLab/beir上公开获取。

英文摘要:

Existing neural information retrieval (IR) models have often been studied in homogeneous and narrow settings, which has considerably limited insights into their out-of-distribution (OOD) generalization capabilities. To address this, and to facilitate researchers to broadly evaluate the effectiveness of their models, we introduce Benchmarking-IR (BEIR), a robust and heterogeneous evaluation benchmark for information retrieval. We leverage a careful selection of 18 publicly available datasets from diverse text retrieval tasks and domains and evaluate 10 state-of-the-art retrieval systems including lexical, sparse, dense, late-interaction and re-ranking architectures on the BEIR benchmark. Our results show BM25 is a robust baseline and re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs. In contrast, dense and sparse-retrieval models are computationally more efficient but often underperform other approaches, highlighting the considerable room for improvement in their generalization capabilities. We hope this framework allows us to better evaluate and understand existing retrieval systems, and contributes to accelerating progress towards better robust and generalizable systems in the future. BEIR is publicly available at https://github.com/UKPLab/beir.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑