Large language models are increasingly used as automated evaluators in research and enterprise settings, a practice known as LLM-as-a-judge. While prior work has examined accuracy, bias, and alignment with human preferences, far less attention has been given to how consistently LLMs assign numerical scores, an important concern for many production workflows. This study systematically evaluates scoring stability across five commonly used models, GPT-4o, GPT-4o-mini, Gemini-2.5-Flash, Claude-Haiku-4.5, and Claude-Sonnet-4.5, two temperature settings, and real enterprise question-answer pairs drawn from a retrieval-augmented generation (RAG) system. We address three questions: how stable a model's scores are across repeated runs, how differently models score identical inputs, and how temperature affects scoring consistency. Temperature controls the determinism of an LLM's output. Despite expectations of stability at temperature=0, we observe substantial variability across models, with completeness scoring showing the largest fluctuations. Cross-model comparisons reveal systematic differences in strictness and interpretive style, leading to divergent ratings for the same answers. Lower temperatures improve stability for some models, notably GPT-4o and Gemini, but have limited or inconsistent effects for Anthropic models. These findings have important implications for enterprise pipelines that rely on LLM-generated scores for routing, triage, gating, or quality control. Identical inputs can receive different scores depending on model, family, or temperature, raising concerns around fairness, reproducibility, and operational reliability. Our results highlight the need for monitoring, robust parsing, and hybrid human-LLM evaluation strategies to ensure dependable use of LLM-as-a-judge in production environments.
Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
Technische Universität Berlin（柏林技术大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫茨研究所）；Marburg University（马尔堡大学）；Centre for European Research in Trusted AI (CERTAIN)（欧洲可信人工智能研究中心）
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
The University of Manchester（曼彻斯特大学）；The University of Sheffield（谢菲尔德大学）；University of Pittsburgh（匹兹堡大学）；Shanghai University of Finance and Economics（上海财经大学）
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Institute of Science Tokyo（东京科学研究院）；Nanjing University（南京大学）；Westlake University（西湖大学）；Southeast University（东南大学）
Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
DIPF | Leibniz Institute for Research and Information in Education（DIPF | 莱布尼茨教育与研究信息研究所）；Centre for International Student Assessment (ZIB)（国际学生评估中心（ZIB））；Faculty of Computer Science, Goethe University Frankfurt（法兰克福大学计算机科学学院）；Chemnitz University of Technology（开姆尼茨工业大学）
Beyond the Surface: Measuring Self-Preference in LLM Judgments
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学首都人工智能学院）；Huawei Poisson Lab（华为Poisson实验室）；Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理重点实验室）
PRISM: Calibrating Black-Box Judges with Counterfactual Probes
Massachusetts Institute of Technology（麻省理工学院）；Sakana AI；Lab for Information and Decision Systems（信息与决策系统实验室）；Statistics and Data Science Center（统计与数据科学中心）；EECS（电气工程与计算机科学系）
