Large language models (LLMs) are widely used as scalable evaluators of model responses in lieu of human annotators. However, imperfect sensitivity and specificity of the LLM judges induce bias in naive evaluation scores. We propose a simple plug-in framework that corrects this bias and enables statistically principled uncertainty quantification. Our framework constructs confidence intervals that account for uncertainty from both the test dataset and a human-labeled calibration dataset. Additionally, it uses an adaptive strategy to allocate calibration samples for tighter intervals. Importantly, we characterize parameter regimes defined by the true evaluation score and the LLM judge's sensitivity and specificity in which our LLM-based evaluation yields more reliable estimates than human-only evaluation. Moreover, we show that our framework remains unbiased under distribution shift between the test and calibration datasets, in contrast to existing approaches.
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU)HiTZ中心);University of Alicante (UA)(阿利坎特大学(UA));GOI Institute, Basque Summer University (UEU)(巴斯克夏季大学(UEU)GOI研究所)
From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
Faculty of Computer Science(计算机科学系);UniVie Doctoral School Computer Science(维也纳大学计算机科学博士学院);Faculty of Philological and Cultural Studies(文学与文化研究系);University of Vienna(维也纳大学);Saarland University(萨尔兰大学);Eastern Institute of Technology(东部技术研究所);Department of Statistics(统计系);LMU Munich(慕尼黑莱茵河大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Pretraining Team, xAI(预训练团队,xAI);Hong Kong University of Science and Technology (GuangZhou)(香港科技大学(广州))
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Dyson School of Design Engineering, Imperial College London(伦敦帝国理工学院戴森设计工程学院);Department of Electrical and Electronic Engineering, Imperial College London(伦敦帝国理工学院电气与电子工程系);School of Electronic Information, Nanchang Institute of Technology(南昌工程学院电子信息学院)
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo(东京大学医学研究生院放射学与生物医学工程系);Department of Radiology, The University of Tokyo Hospital(东京大学医院放射科);Department of Radiology, Kanazawa University Graduate School of Medical Sciences(金泽大学医学研究生院放射科);Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital(东京大学医院计算诊断放射学与预防医学系);Department of Radiology, School of Medicine, Jichi Medical University(自治医科大学医学院放射科)
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
AMAP, Alibaba Group(阿里巴巴集团AMAP实验室);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Southern California(南加州大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)
Manacá-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge
National Laboratory for Scientific Computing (LNCC)(国家科学计算实验室);Institute of Artificial Intelligence(人工智能研究所)
How do Humans Process AI-generated Hallucination Contents: a Neuroimaging Study
Department of Computer Science and Technology, Tsinghua University, Beijing, China(清华大学计算机科学与技术系);Institute of Trustworthy Embodied AI, Fudan University, Shanghai, China(复旦大学可信具身人工智能研究院)
ReportLogic: Evaluating Logical Quality in Deep Research Reports
Leiden Institute of Advanced Computer Science, Leiden University(莱顿先进计算机科学研究所,莱顿大学);Ant Group(蚂蚁集团);CISPA Helmholtz Center for Information Security(信息安全霍普夫中心)