High-quality relevance judgements over large query sets are essential for evaluating Information Retrieval (IR) systems, yet manual annotation remains costly and time-consuming. Large Language Models (LLMs) have recently shown promise as automatic relevance assessors, but their reliability is still limited. Most existing approaches rely on zero-shot prompting or In-Context Learning (ICL) with a small number of labeled examples. However, standard ICL treats examples as independent instances and fails to explicitly capture the underlying relevance criteria of a topic, restricting its ability to generalize to unseen query-document pairs. To address this limitation, we introduce Relevance Context Learning (RCL), a novel framework that leverages human relevance judgements to explicitly model topic-specific relevance criteria. Rather than directly using labeled examples for in-context prediction, RCL first prompts an LLM (Instructor LLM) to analyze sets of judged query-document pairs and generate explicit narratives that describe what constitutes relevance for a given topic. These relevance narratives are then used as structured prompts to guide a second LLM (Assessor LLM) in producing relevance judgements. To evaluate RCL in a realistic data collection setting, we propose a hybrid pooling strategy in which a shallow depth-\textit{k} pool from participating systems is judged by human assessors, while the remaining documents are labeled by LLMs. Experimental results demonstrate that RCL substantially outperforms zero-shot prompting and consistently improves over standard ICL. Overall, our findings indicate that transforming relevance examples into explicit, context-aware relevance narratives is a more effective way of exploiting human judgements for LLM-based IR dataset construction.
In the context of depth-$k$ pooling for constructing web search test collections, we compare two approaches to ordering pooled documents for relevance assessors: the prioritisation strategy (PRI) used widely at NTCIR, and the simple randomisation strategy (RND). In order to address research questions regarding PRI and RND, we have constructed and released the WWW3E8 data set, which contains eight independent relevance labels for 32,375 topic-document pairs, i.e., a total of 259,000 labels. Four of the eight relevance labels were obtained from PRI-based pools; the other four were obtained from …
How Relevance Emerges: Interpreting LoRA Fine-Tuning in Reranking LLMs
Center for Intelligent Information Retrieval, Manning College of Information and Computer Sciences, University of Massachusetts Amherst（智能信息检索中心，信息与计算机科学学院，马萨诸塞大学阿姆赫斯特分校）
An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
How role-play shapes relevance judgment in zero-shot LLM rankers
Leiden Institute of Advanced Computer Science, Leiden University（莱顿大学高级计算机科学研究所）；Center for Language and Cognition, University of Groningen（格罗宁根大学语言与认知中心）；Brown University（布朗大学）；University of Amsterdam（阿姆斯特丹大学）
Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R$^2$)GRPO
HKUST Hong Kong SAR, China（香港科技大学）；SEU Jiangsu, China（江苏大学）；HKUST(GZ) Guangzhou, China（香港科技大学（广州））；Zhipu AI Beijing, China（智谱AI）；HKUST(GZ), HKUST Guangzhou, China（香港科技大学（广州）、香港科技大学）
Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学耿丽人工智能学院）；Beijing Normal University（北京师范大学）；DataArc Tech Ltd.（数据弧科技有限公司）；Huawei Cloud BU（华为云业务单元）
REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；National University of Singapore（新加坡国立大学）；Baidu Inc.（百度公司）；School of Information, Renmin University of China（中国人民大学信息学院）
