Large Language Models (LLMs) have been used as relevance assessors for Information Retrieval (IR) evaluation collection creation due to reduced cost and increased scalability as compared to human assessors. While previous research has looked at the reliability of LLMs as compared to human assessors, in this work, we aim to understand if LLMs make systematic mistakes when judging relevance, rather than just understanding how good they are on average. To this aim, we propose a novel representational method for queries and documents that allows us to analyze relevance label distributions and compare LLM and human labels to identify patterns of disagreement and localize systematic areas of disagreement. We introduce a clustering-based framework that embeds query-document (Q-D) pairs into a joint semantic space, treating relevance as a relational property. Experiments on TREC Deep Learning 2019 and 2020 show that systematic disagreement between humans and LLMs is concentrated in specific semantic clusters rather than distributed randomly. Query-level analyses reveal recurring failures, most often in definition-seeking, policy-related, or ambiguous contexts. Queries with large variation in agreement across their clusters emerge as disagreement hotspots, where LLMs tend to under-recall relevant content or over-include irrelevant material. This framework links global diagnostics with localized clustering to uncover hidden weaknesses in LLM judgments, enabling bias-aware and more reliable IR evaluation.
An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学耿丽人工智能学院）；Beijing Normal University（北京师范大学）；DataArc Tech Ltd.（数据弧科技有限公司）；Huawei Cloud BU（华为云业务单元）
DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；Swiss Finance Institute (SFI)（瑞士金融研究所）；University of Electronic Science and Technology of China（电子科技大学）
Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval
University of Science and Technology of China（中国科学技术大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Hong Kong Polytechnic University（香港理工大学）；Hong Kong University of Science and Technology（香港科技大学）
From Judgments to Issues: Structured Extraction of Legal Reasoning with Citation-Hallucination Control
Quantitative and Digital Law Laboratory（量化与数字法律实验室）；CIRSFID - Alma AI, Faculty of Law University of Bologna（CIRSFID - 阿尔玛AI，博洛尼亚大学法学院）；Robert Schuman Centre European University Institute（罗伯特·施曼中心欧洲大学研究所）
If Probable, Then Acceptable? Understanding Conditional Acceptability Judgments in Large Language Models
MaiNLP, Center for Information and Language Processing, LMU Munich（MaiNLP，信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML), Munich（慕尼黑机器学习中心（MCML），慕尼黑）
North China Electric Power University（华北电力大学）；Renmin University of China（中国人民大学）
A previous similar case in common law systems can be used as a reference with respect to the current case such that identical situations can be treated similarly in every case. However, current approaches for judgment document similarity computation failed to capture the core semantics of judgment documents and therefore suffer from lower accuracy and higher computation complexity. In this paper, a knowledge block summarization based machine learning approach is proposed to compute the semantic similarity of Chinese judgment documents. By utilizing domain ontologies for judgment documents, the…
Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking
University of Science and Technology of China（中国科学技术大学）；Yuanbao Team, Tencent（腾讯元宝团队）；University of Chinese Academy of Sciences（中国科学院大学）；Shandong University（山东大学）
Learning to Rank (LETOR) algorithms are usually trained on annotated corpora where a single relevance label is assigned to each available document-topic pair. Within the Cranfield framework, relevance labels result from merging either multiple expertly curated or crowdsourced human assessments. In this paper, we explore how to train LETOR models with relevance judgments distributions (either real or synthetically generated) assigned to document-topic pairs instead of single-valued relevance labels. We propose five new probabilistic loss functions to deal with the higher expressive power provid…
Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of experts' relevance judgment results are usually taken as ground truth to "objectively" evaluate the performance of the IR systems. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the potentially biased judgment results stemmed from using only a single expert's judgment. Nevertheless, different judges may have different opinions and may not agree with each other, and the inconsistency…
Relevance judgment in Information Retrieval is influenced by multiple factors. These include not only the topicality of the documents but also other user oriented factors like trust, user interest, etc. Recent works have identified these various factors into seven dimensions of relevance. In a previous work, these relevance dimensions were quantified and user's cognitive state with respect to a document was represented as a state vector in a Hilbert Space, with each relevance dimension representing a basis. It was observed that relevance dimensions are incompatible in some documents, when maki…
There is a growing body of research which has investigated relevance judgment in IR being influenced by multiple factors or dimensions. At the same time, the Order Effects in sequential decision making have been quantitatively detected and studied in Mathematical Psychology. Combining the two phenomena, there have been some user studies carried out which investigate the Order Effects and thus incompatibility in different dimensions of relevance. In this work, we propose a methodology for carrying out such an investigation in large scale and real world data using query logs of a web search engi…
In this paper we study the effect on crowd worker efficiency and effectiveness of the dominance of one class in the data they process. We aim at understanding if there is any positive or negative bias in workers seeing many negative examples in the identification of positive labels. To test our hypothesis, we design an experiment where crowd workers are asked to judge the relevance of documents presented in different orders. Our findings indicate that there is a significant improvement in the quality of relevance judgements when presenting relevant results before the non-relevant ones.
