重排序器作为相关性判断者
Re-Rankers as Relevance Judges
AI总结：
本文提出将重排序器作为相关性判断者的方法，通过两种适应策略在重排序任务中实现优于现有方法的性能。
摘要：
利用大型语言模型（LLMs）来预测相关性判断已显示出有前景的结果。大多数研究将此任务视为一个独立的研究领域，例如专注于提示设计以根据查询和段落预测相关性标签。然而，预测相关性判断本质上是一种相关性预测，这是在重排序任务中广泛研究的问题。尽管存在这种潜在的重叠，很少有研究探索重用或适应已建立的重排序方法来预测相关性判断，导致潜在的资源浪费和重复开发。为了填补这一空白，我们采用重排序器作为相关性判断者的设置来重现重排序器。我们设计了两种适应策略：（i）使用由重排序器生成的二进制标记（例如“true”和“false”）作为直接判断，以及（ii）通过阈值处理将连续的重排序分数转换为二进制标签。我们在TREC-DL 2019到2023年间进行了广泛的实验，使用了8个来自3个家族的重排序器，从220M到32B不等，并分析了基于重排序器的判断者的评估偏差。结果表明，基于重排序器的相关性判断者在两种策略下，可以在大约40%到50%的情况下优于UMBRELA，一个最先进的基于LLM的相关性判断者；它们还表现出对自身和同一家族重排序器的强烈自我偏好，以及跨家族偏见。
英文摘要：
Using large language models (LLMs) to predict relevance judgments has shown promising results. Most studies treat this task as a distinct research line, e.g., focusing on prompt design for predicting relevance labels given a query and passage. However, predicting relevance judgments is essentially a form of relevance prediction, a problem extensively studied in tasks such as re-ranking. Despite this potential overlap, little research has explored reusing or adapting established re-ranking methods to predict relevance judgments, leading to potential resource waste and redundant development. To bridge this gap, we reproduce re-rankers in a re-ranker-as-relevance-judge setup. We design two adaptation strategies: (i) using binary tokens (e.g., "true" and "false") generated by a re-ranker as direct judgments, and (ii) converting continuous re-ranking scores into binary labels via thresholding. We perform extensive experiments on TREC-DL 2019 to 2023 with 8 re-rankers from 3 families, ranging from 220M to 32B, and analyse the evaluation bias exhibited by re-ranker-based judges. Results show that re-ranker-based relevance judges, under both strategies, can outperform UMBRELA, a state-of-the-art LLM-based relevance judge, in around 40% to 50% of the cases; they also exhibit strong self-preference towards their own and same-family re-rankers, as well as cross-family bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
