Relevance judgments are central to the evaluation of Information Retrieval (IR) systems, but obtaining them from human annotators is costly and time-consuming. Large Language Models (LLMs) have recently been proposed as automated assessors, showing promising alignment with human annotations. Most prior studies have treated documents as fixed units, feeding their full content directly to LLM assessors. We investigate how text summarization affects the reliability of LLM-based judgments and their downstream impact on IR evaluation. Using state-of-the-art LLMs across multiple TREC collections, we compare judgments made from full documents with those based on LLM-generated summaries of different lengths. We examine their agreement with human labels, their effect on retrieval effectiveness evaluation, and their influence on IR systems' ranking stability. Our findings show that summary-based judgments achieve comparable stability in systems' ranking to full-document judgments, while introducing systematic shifts in label distributions and biases that vary by model and dataset. These results highlight summarization as both an opportunity for more efficient large-scale IR evaluation and a methodological choice with important implications for the reliability of automatic judgments.
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU）HiTZ中心）；University of Alicante (UA)（阿利坎特大学（UA））；GOI Institute, Basque Summer University (UEU)（巴斯克夏季大学（UEU）GOI研究所）
An Iterative Utility Judgment Framework Inspired by Philosophical Relevance via LLMs
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学耿丽人工智能学院）；Beijing Normal University（北京师范大学）；DataArc Tech Ltd.（数据弧科技有限公司）；Huawei Cloud BU（华为云业务单元）
A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval
A.K.Choudhury School of IT（A.K.Choudhury信息科技学院）；University of Calcutta（加尔各答大学）；Department of Computer Science（计算机科学系）；Bangabasi Morning College（Bangabasi晨学院）
References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluation
MaiNLP, LMU Munich（LMU慕尼黑人工智能实验室）；Munich Center for Machine Learning (MCML), Germany（慕尼黑机器学习中心）；Department of Linguistics, University of Pittsburgh, USA（匹兹堡大学语言学系）；NLP Group, Department of Information and Computing Sciences, Utrecht University, Netherlands（乌得勒支大学信息与计算科学系自然语言处理组）
