随着大型语言模型（LLM）越来越多地被考虑用于自动评估和反馈，理解LLM评分何时有效至关重要。我们评估了LLM作为评判者在三种物理评估格式——结构化问题、书面论文和科学图表——中的评分表现，比较了GPT-5.2、Grok 4.1、Claude Opus 4.5、DeepSeek-V3.2、Gemini Pro 3以及委员会聚合在盲评、提供解决方案、错误解决方案和锚定条件下的结果。我们区分了绝对准确性和排序一致性，因为评分系统可以匹配人类评分的分布，但未能按质量对回答进行排序。在不同任务类型中，性能表现出强烈的任务依赖性。对于盲评大学考试问题（n=771）以及中学和大学结构化问题（n=1151），模型与人类评分者显示出稳健的排序一致性（Spearman ρ > 0.6），官方解决方案减少了误差并加强了一致性。错误解决方案降低了绝对准确性，表明模型依赖于提供的参考，但排序一致性保持不变。论文评分的行为根本不同。在n=55份答卷（n=275篇论文）中，盲评AI评分比人类评分更严厉且变异性更大，添加评分方案并未改善排序一致性。锚定范例将AI均值移近人类均值，并将方差压缩到低于人类标准差，但排序一致性仍接近零。对于基于代码的图表元素（n=1400），模型实现了高排序一致性（ρ > 0.84）和近乎线性的校准。在所有任务类型中，有效性取决于评估任务的结构——即评分可以映射到明确、可观察的评分特征的程度——以及人类基准的可靠性，而非原始模型能力。
英文摘要：
As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking is valid is essential. We evaluate LLM-as-a-judge marking across four settings spanning three assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini 3 Pro, and committee aggregations against human markers under blind, solution-provided, false-solution, and anchored conditions. We distinguish absolute accuracy from rank-order agreement, since a marking system can match the distribution of human marks while failing to order responses by quality. Across task types, performance is sharply task-dependent. For blind university exam questions ($n=771$) and secondary and university structured questions ($n=1151$), models show robust rank-order agreement with human markers (Spearman $ρ> 0.6$), with official solutions reducing error and strengthening agreement. False solutions degrade absolute accuracy, showing that models defer to provided references, but leave rank-ordering intact. Essay marking behaves fundamentally differently. Across $n=55$ scripts, each containing five essays ($n=275$ essays total), blind AI marking is harsher and more variable than human marking and adding marking guidance does not improve rank-order agreement. Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but rank-order agreement remains near-zero. For code-based plot elements ($n=1400$), models achieve high rank-order agreement ($ρ> 0.83$) with near-linear calibration. LLM marking validity depended strongly on the assessment task, and this held for every contemporary model tested. The results also show that the reliability of the human benchmark constrains the claims that can be made about AI-human agreement.
Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
Technische Universität Berlin（柏林技术大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫茨研究所）；Marburg University（马尔堡大学）；Centre for European Research in Trusted AI (CERTAIN)（欧洲可信人工智能研究中心）
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Dyson School of Design Engineering, Imperial College London（伦敦帝国理工学院戴森设计工程学院）；Department of Electrical and Electronic Engineering, Imperial College London（伦敦帝国理工学院电气与电子工程系）；School of Electronic Information, Nanchang Institute of Technology（南昌工程学院电子信息学院）
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
DIPF | Leibniz Institute for Research and Information in Education（DIPF | 莱布尼茨教育与研究信息研究所）；Centre for International Student Assessment (ZIB)（国际学生评估中心（ZIB））；Faculty of Computer Science, Goethe University Frankfurt（法兰克福大学计算机科学学院）；Chemnitz University of Technology（开姆尼茨工业大学）
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)（国立信息学研究所）；University of Amsterdam（阿姆斯特丹大学）；University of Tokyo（东京大学）；Hitotsubashi University（一桥大学）；Centrum Wiskunde & Informatica (CWI)（荷兰数学与计算机科学研究中心）；Utrecht University（乌得勒支大学）
LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
University of Amsterdam（阿姆斯特丹大学）；University of Trento（特伦托大学）；University of Copenhagen（哥本哈根大学）；Utrecht University（乌得勒支大学）；Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）；ETH Zürich（苏黎世联邦理工学院）；Saarland University（萨尔大学）；Universidade de Lisboa & Unbabel（里斯本大学及Unbabel）；LMU Munich & MCML（慕尼黑大学及MCML）；University of Potsdam（波茨坦大学）；Heriot-Watt University（赫瑞线学院）
Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals
Quality and Usability Lab, Technische Universität Berlin（柏林技术大学质量与可用性实验室）；University of Marburg（马尔堡大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Deutsche Telekom（德国电信）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Department of Biomedical Informatics, School of Medicine, University of Pittsburgh（匹兹堡大学医学院生物医学信息学系）；Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh（匹兹堡大学健康与康复科学学院健康信息管理系）；OpenCura, Health Innovation Consortium（OpenCura健康创新联盟）；Northwestern University, Kellogg School of Management（西北大学凯洛格管理学院）；Intelligent Systems Program, School of Computing and Information, University of Pittsburgh（匹兹堡大学计算与信息学院智能系统项目）；Johns Hopkins University School of Medicine Biomedical Informatics and Data Science（约翰霍普金斯大学医学院生物医学信息学与数据科学）；Clinical and Translational Science Institute, University of Pittsburgh（匹兹堡大学临床与转化科学研究所）；Institute for Biomedical Informatics, University of Pennsylvania（宾夕法尼亚大学生物医学信息学研究所）；Data Science, School of Computing and Information, University of Pittsburgh（匹兹堡大学计算与信息学院数据科学）
HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Pretraining Team, xAI（预训练团队，xAI）；Hong Kong University of Science and Technology (GuangZhou)（香港科技大学（广州））
How Output Format Confounds Data Quality and Capability in Instruction Tuning
University of Chinese Academy of Sciences（中国科学院大学）；Pusan National University（釜山国立大学）；Shenzhen University of Advanced Technology（深圳先进技术大学）；Monash University（莫纳什大学）
Monotonic Reference-Free Refinement for Autoformalization
Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；School of Computer Science, University of Sheffield（谢菲尔德大学计算机科学学院）；Idiap Research Institute（Idiap研究所）；National Biomarker Centre, CRUK Manchester Institute（国家生物标记中心，CRUK曼彻斯特研究所）
Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judge
Business School, Shandong University of Technology（山东理工大学商学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Guangdong Institute of Intelligent Science and Technology（广东智能科学与技术研究院）；Macau Millennium College（澳门 millennium 学院）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo（东京大学医学研究生院放射学与生物医学工程系）；Department of Radiology, The University of Tokyo Hospital（东京大学医院放射科）；Department of Radiology, Kanazawa University Graduate School of Medical Sciences（金泽大学医学研究生院放射科）；Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital（东京大学医院计算诊断放射学与预防医学系）；Department of Radiology, School of Medicine, Jichi Medical University（自治医科大学医学院放射科）
Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation
Manipal University Jaipur（马纳普大学贾普尔分校）；Indian Institute of Technology, Patna（印度理工学院帕坦分校）；Indian Institute of Science Education and Research, Kolkata（印度科学教育与研究学院科钦分校）
