Large language models (LLMs) are being widely applied across various fields, but as tasks become more complex, evaluating their responses is increasingly challenging. Compared to human evaluators, the use of LLMs to support performance evaluation offers a more efficient alternative. However, most studies focus mainly on aligning LLMs' judgments with human preferences, overlooking the existence of biases and mistakes in human judgment. Furthermore, how to select suitable LLM judgments given multiple potential LLM responses remains underexplored. To address these two aforementioned issues, we propose a three-stage meta-judge selection pipeline: 1) developing a comprehensive rubric with GPT-4 and human experts, 2) using three advanced LLM agents to score judgments, and 3) applying a threshold to filter out low-scoring judgments. Compared to methods using a single LLM as both judge and meta-judge, our pipeline introduces multi-agent collaboration and a more comprehensive rubric. Experimental results on the JudgeBench dataset show about 15.55\% improvement compared to raw judgments and about 8.37\% improvement over the single-agent baseline. Our work demonstrates the potential of LLMs as meta-judges and lays the foundation for future research on constructing preference datasets for LLM-as-a-judge reinforcement learning.
When Trust Meets Truth: Trust-Truth Separability in LLM-as-Judge
National Institute of Informatics (NII)（信息学研究所（NII））；University of Amsterdam（阿姆斯特丹大学）；University of Tokyo（东京大学）；Vrije Universiteit Amsterdam（阿姆斯特丹自由大学）；Centrum Wiskunde & Informatica (CWI)（数学和计算机科学中心（CWI））；Utrecht University（乌得勒支大学）
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Arizona State University（亚利桑那州立大学）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）；University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；Northwestern University（西北大学）；University of California, Berkeley（加州大学伯克利分校）；Emory University（埃默里大学）
Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions
Nanyang Technological University（南洋理工大学）；National University of Singapore（新加坡国立大学）；CFAR, Agency for Science, Technology and Research（新加坡科技研究局高等研究院）；UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；Carnegie Mellon University（卡内基梅隆大学）；IHPC, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究院）
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)（巴斯克大学（UPV/EHU）HiTZ中心）；University of Alicante (UA)（阿利坎特大学（UA））；GOI Institute, Basque Summer University (UEU)（巴斯克夏季大学（UEU）GOI研究所）
From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
Faculty of Computer Science（计算机科学系）；UniVie Doctoral School Computer Science（维也纳大学计算机科学博士学院）；Faculty of Philological and Cultural Studies（文学与文化研究系）；University of Vienna（维也纳大学）；Saarland University（萨尔兰大学）；Eastern Institute of Technology（东部技术研究所）；Department of Statistics（统计系）；LMU Munich（慕尼黑莱茵河大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Lanzhou University（兰州大学）；Shandong University School of Literature（山东大学文学院）
One of the important topics in the research field of Chinese classical poetry is to analyze the poetic style. By examining the relevant works of previous dynasties, researchers judge a poetic style mostly by their subjective feelings, and refer to the previous evaluations that have become a certain conclusion. Although this judgment method is often effective, there may be some errors. This paper builds the most perfect data set of Chinese classical poetry at present, trains a BART-poem pre -trained model on this data set, and puts forward a generally applicable poetry style judgment method bas…
When Is It Acceptable to Break the Rules? Knowledge Representation of Moral Judgement Based on Empirical Data
University of Exeter（埃克塞特大学）；Massachusetts Institute of Technology（麻省理工学院）；Harvard University（哈佛大学）；European University Institute（欧洲大学学院）；Tulane University（杜兰大学）；Max Planck Institute for Human Development（马克斯·普朗克人类发展研究所）；IBM Research（IBM研究院）
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)（国立信息学研究所）；University of Amsterdam（阿姆斯特丹大学）；University of Tokyo（东京大学）；Hitotsubashi University（一桥大学）；Centrum Wiskunde & Informatica (CWI)（荷兰数学与计算机科学研究中心）；Utrecht University（乌得勒支大学）
Automatic judgment prediction aims to predict the judicial results based on case materials. It has been studied for several decades mainly by lawyers and judges, considered as a novel and prospective application of artificial intelligence techniques in the legal field. Most existing methods follow the text classification framework, which fails to model the complex interactions among complementary case materials. To address this issue, we formalize the task as Legal Reading Comprehension according to the legal scenario. Following the working protocol of human judges, LRC predicts the final judg…
Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
Department of Computer Science and Information Engineering（计算机科学与信息工程系）；National Taiwan University（国立台湾大学）；Institute of Information Science, Academia Sinica（学术院信息研究所）；AI Research Center (AINTU)（人工智能研究中心）
Beyond Correctness: Enhancing Architectural Reasoning in Code LLMs via Scalable Labeling with Agentic Judgment
Centre for Software Excellence, Huawei Canada（华为加拿大软件卓越中心）；Department of Computer Science, University of Manitoba, Canada（曼尼托巴大学计算机科学系）；School of Computing, Queen’s University, Canada（皇后大学计算科学学院）
When Wording Steers the Evaluation: Framing Bias in LLM judges
IPAI, Seoul National University（IPAI，首尔国立大学）；Dept. of ECE, Seoul National University（电子工程系，首尔国立大学）；LG AI Research（LG人工智能研究）；SNU-LG AI Research Center（SNU-LG人工智能研究中心）
Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages
LT3, IDLab, Universiteit Gent, Barcelona Supercomputing Center, LMU Munich & Munich Center for Machine Learning, German Center for Addiction Research in Childhood and Adolescence, University Medical Center Hamburg-Eppendorf, Mila - Quebec AI Institute, McGill University, Canada CIFAR AI Chair（LT3、IDLab、根特大学、巴塞罗那超级计算中心、慕尼黑莱茵河大学及慕尼黑机器学习中心、德国成年期成瘾研究中心、汉堡埃彭多夫大学医学中心、魁北克人工智能研究所、麦吉尔大学、加拿大 CIFAR 人工智能主席）
