As Large Language Models (LLMs) become integrated into high-stakes domains, there is a growing need for evaluation methods that are both scalable for real-time deployment and reliable for critical decision-making. While human evaluation is reliable, it is slow and costly. Single LLM judges are biased, and static juries lack adaptability. To overcome these limitations, we propose LLM Jury-on-Demand - a dynamic, learning-based framework for scalable and context-aware evaluation. Our method trains a set of reliability predictors to assess when LLM judges will agree with human experts, leveraging token distributions, embeddings, and structural input features. This enables a fully adaptive evaluation where, for each data point, an optimal jury of the most reliable judges is dynamically selected, and their scores are aggregated using their reliability as weights. Experiments on summarization and RAG benchmarks show that our dynamic jury system achieves significantly higher correlation with human judgment than both single-judge and static-jury baselines. These results highlight the promise of adaptive, learning-based juries for building scalable, more reliable and trustworthy evaluation systems for modern LLMs in high-stakes domains.
Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
Apart Research;Odyssean Institute;London School of Economics and Political Science(伦敦政治经济学院);Stanford Institute for Human-Centered AI(斯坦福大学以人为本人工智能研究所)
An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
Singapore Management University(新加坡国立管理学院);DGIST(韩国高等智能科学研究院);Monash University(墨尔本大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Huawei Software Engineering Application Technology Lab(华为软件工程应用技术实验室);Zhejiang University(浙江大学)
Jury Duty: Calibration and Orientation Failures in MLLM-as-a-Judge Under Cultural Ambiguity
Salesforce AI Research(Salesforce AI 研究院);University of Cambridge(剑桥大学);University of Colorado Boulder(科罗拉多大学博尔德分校);Carnegie Mellon University(卡内基梅隆大学);UIUC(伊利诺伊大学厄巴纳-香槟分校)
Can LLMs Accurately Score Medical Diagnoses and Clinical Reasoning?
Wits MIND Institute, University of the Witwatersrand, Johannesburg, South Africa(维特士心理研究所,沃斯兰德大学,约翰内斯堡,南非);Grai Labs, Cape Town, South Africa(格雷实验室,开普敦,南非);South African Medical Research Council Vaccines and Infectious Diseases Analytics Research Unit, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa(南非医学研究理事会疫苗和传染病分析研究组,健康科学学院,沃斯兰德大学,约翰内斯堡,南非);Department of Internal Medicine, Charlotte Maxeke Johannesburg Academic Hospital, and Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa(内科学系,查理·马克斯凯约翰内斯堡学术医院,以及健康科学学院,沃斯兰德大学,约翰内斯堡,南非);Department of Paediatrics and Child Health, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa(儿科学与儿童健康系,健康科学学院,沃斯兰德大学,约翰内斯堡,南非);Wits MIND Institute, University of the Witwatersrand, Johannesbu(维特士心理研究所,沃斯兰德大学,约翰内斯堡)
A Multi-Stage Validation Framework for Trustworthy Large-scale Clinical Information Extraction using Large Language Models
Oak Ridge National Laboratory(橡树岭国家实验室);Program Evaluation and Resource Center, Office of Mental Health and Office of Suicide Prevention, Department of Veterans Affairs(退伍军人事务部心理健康办公室和自杀预防办公室项目评估与资源中心);Vanderbilt University Medical Center(范德比尔特大学医学中心);VA Maryland Health Care System(退伍军人事务部马里兰医疗保健系统);VA Desert Pacific Healthcare Network(退伍军人事务部沙漠太平洋医疗网络);VA Connecticut Health Care System(退伍军人事务部康涅狄格医疗保健系统);Yale School of Medicine(耶鲁医学院)
Hybrid Retrieval-Augmented Generation Agent for Trustworthy Legal Question Answering in Judicial Forensics
Department of Electronic Engineering, City University of Hong Kong(DongGuan)(香港城市大学(东莞)电子工程系);Department of Electronic Engineering, City University of Hong Kong(香港城市大学电子工程系)
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
Peking University(北京大学);National University of Singapore(新加坡国立大学);Institute of Science Tokyo(东京科学研究院);Nanjing University(南京大学);Westlake University(西湖大学);Southeast University(东南大学)
Meta-Judging with Large Language Models: Concepts, Methods, and Challenges
University of Coimbra, CISUC/LASI – Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering(科英布拉大学,CISUC/LASI——科英布拉大学信息与系统研究中心,信息工程系);Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, Portugal(科英布拉理工大学)
The Fellowship of the LLMs: Multi-Model Workflows for Synthetic Preference Optimization Dataset Generation
Lahore University of Management Sciences(拉合尔管理科学大学);University of Michigan - Ann Arbor(密歇根大学安娜堡分校);EMBL European Bioinformatics Institute(欧洲生物信息学研究所);Strategize Inc(Strategize公司)
CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
College of Computer Science and Technology(计算机科学与技术学院);School of Artificial Intelligence and Computer Science(人工智能与计算机科学学院);School of Software(软件学院);School of Software Technology(软件技术学院);School of Computing and Information Systems(计算与信息系统学院);School of Computing and Mathematical Sciences(计算与数学科学学院)
Developing and evaluating a chatbot to support maternal health care
Carnegie Mellon University(卡内基梅隆大学);Population Council Institute(人口理事会研究所);Sitaram Bhartia Institute of Science and Research(西塔拉姆·巴提亚科学与研究研究所);Nivi, Inc.(Nivi公司)
FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
Jilin University(吉林大学);University of Central Florida(中佛罗里达大学);Arizona State University(亚利桑那州立大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);City University of Hong Kong(香港城市大学)
Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);Graduate College for Engineers, Beijing University of Posts and Telecommunications(北京邮电大学研究生院工程师学院);School of Mathematical Sciences, Fudan University(复旦大学数学科学学院);School of Cyberspace Security, Beijing University of Posts and Telecommunications(北京邮电大学网络空间安全学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);Chu Kochen Honors College, Zhejiang University(浙江大学竺可桢学院);Department of Psychological and Cognitive Sciences, Tsinghua University(清华大学心理学与认知科学系);State Key Laboratory of Virtual Reality Technology and Systems, Beihang University(北京航空航天大学虚拟现实技术与系统国家重点实验室);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)