arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.09947cs.IR
收藏

神经IR查询性能预测:我们到了吗?

Query Performance Prediction for Neural IR: Are We There Yet?

Guglielmo Faggioli, Thibault Formal, Stefano Marchesin, Stéphane Clinchant, Nicola Ferro, Benjamin Piwowarski

摘要:

信息检索中的评估依赖于事后经验程序,这是耗时且昂贵的操作。为了缓解这一点,查询性能预测(QPP)模型已经开发出来,以估计系统的性能,而不需要人为的相关性判断。这些模型通常依赖于查询和语料库的词汇特征,已被应用于传统的稀疏IR方法-取得了不同程度的成功。随着神经信息检索和大型预训练语言模型的出现,检索范式已显著转向更多的语义信号。在这项工作中,我们研究和分析在何种程度上目前的QPP模型可以预测这样的系统的性能。我们的实验考虑了七种传统的词袋和七种基于BERT的IR方法,以及在两个集合深度学习'19和Robust '04上评估的十九种最先进的QPP。我们的研究结果表明,QPP在神经IR系统上的表现在统计学上显著更差。在语义信号突出的环境中(例如,段落检索),与词袋方法相比,它们在神经模型上的性能下降了10%。最重要的是,在面向词汇的场景中,QPP无法预测神经IR系统在与传统方法最不同的查询上的性能。

英文摘要:

Evaluation in Information Retrieval relies on post-hoc empirical procedures, which are time-consuming and expensive operations. To alleviate this, Query Performance Prediction (QPP) models have been developed to estimate the performance of a system without the need for human-made relevance judgements. Such models, usually relying on lexical features from queries and corpora, have been applied to traditional sparse IR methods - with various degrees of success. With the advent of neural IR and large Pre-trained Language Models, the retrieval paradigm has significantly shifted towards more semantic signals. In this work, we study and analyze to what extent current QPP models can predict the performance of such systems. Our experiments consider seven traditional bag-of-words and seven BERT-based IR approaches, as well as nineteen state-of-the-art QPPs evaluated on two collections, Deep Learning '19 and Robust '04. Our findings show that QPPs perform statistically significantly worse on neural IR systems. In settings where semantic signals are prominent (e.g., passage retrieval), their performance on neural models drops by as much as 10% compared to bag-of-words approaches. On top of that, in lexical-oriented scenarios, QPPs fail to predict performance for neural IR systems on those queries where they differ from traditional approaches the most.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑