使用大语言模型生成的相关性判断进行查询性能预测
Query Performance Prediction using Relevance Judgments Generated by Large Language Models
AI总结：
提出QPP-GenRE框架，利用大语言模型生成相关性判断，分解QPP任务以预测多种IR指标并增强可解释性，通过近似策略和微调解决计算与性能挑战，在多个数据集上达到最先进效果。
摘要：
查询性能预测（QPP）旨在无需人工相关性判断的情况下，估计搜索系统针对某个查询的检索质量。以往的QPP方法通常返回单个标量值，且不要求预测值近似于特定的信息检索（IR）评估指标，这导致了一些缺陷：（i）单个标量不足以准确表示不同的IR评估指标，尤其是当指标之间相关性不高时；（ii）单个标量限制了QPP方法的可解释性，因为仅使用标量不足以解释QPP结果。为解决这些问题，我们提出了一种使用自动生成的相关性判断的QPP框架（QPP-GenRE），该框架将QPP分解为独立的子任务，即预测排序列表中每个条目与给定查询的相关性。这使得我们可以使用生成的相关性判断作为伪标签来预测任何IR评估指标。这也使我们能够解释预测的IR评估指标，并识别、跟踪和纠正生成的相关性判断中的错误，以提高QPP质量。我们通过使用开源大语言模型（LLMs）来预测条目的相关性，以确保科学可复现性。我们面临两个主要挑战：（i）在预测考虑召回率的指标时，判断整个语料库的计算成本过高；（ii）在零样本/少样本方式下提示开源LLMs的性能有限。为解决这些挑战，我们设计了一种近似策略来预测考虑召回率的IR指标，并提出使用人工标注的相关性判断对开源LLMs进行微调。在TREC 2019至2022深度学习赛道以及CAsT-19和20数据集上的实验表明，QPP-GenRE在词汇和神经排序器上均实现了最先进的QPP质量。
英文摘要：
Query performance prediction (QPP) aims to estimate the retrieval quality of a search system for a query without human relevance judgments. Previous QPP methods typically return a single scalar value and do not require the predicted values to approximate a specific information retrieval (IR) evaluation measure, leading to certain drawbacks: (i) a single scalar is insufficient to accurately represent different IR evaluation measures, especially when metrics do not highly correlate, and (ii) a single scalar limits the interpretability of QPP methods because solely using a scalar is insufficient to explain QPP results. To address these issues, we propose a QPP framework using automatically generated relevance judgments (QPP-GenRE), which decomposes QPP into independent subtasks of predicting the relevance of each item in a ranked list to a given query. This allows us to predict any IR evaluation measure using the generated relevance judgments as pseudo-labels. This also allows us to interpret predicted IR evaluation measures, and identify, track and rectify errors in generated relevance judgments to improve QPP quality. We predict an item's relevance by using open-source large language models (LLMs) to ensure scientific reproducibility. We face two main challenges: (i) excessive computational costs of judging an entire corpus for predicting a metric considering recall, and (ii) limited performance in prompting open-source LLMs in a zero-/few-shot manner. To solve the challenges, we devise an approximation strategy to predict an IR measure considering recall and propose to fine-tune open-source LLMs using human-labeled relevance judgments. Experiments on the TREC 2019 to 2022 deep learning tracks and CAsT-19 and 20 datasets show that QPP-GenRE achieves state-of-the-art QPP quality for both lexical and neural rankers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
