Large language models (LLMs) are widely used as automated judges, yet most alignment methods target aggregated human judgments and overlook variation among individual annotators. Existing approaches to matching the full distribution of human ratings mainly rely on output-level calibration or parameter fine-tuning. We propose Human-Residual Steering (HRS), to our knowledge the first method to align frozen LLM judges with human judgment distributions through learned hidden-state interventions. An oracle analysis shows that low-rank corrections at a single layer remove most of the discrepancy between judge and human distributions. HRS learns to predict such corrections from the judge's hidden representation and rating logits, training on individual annotations with proper scoring rules for ordinal and nominal judgments and optimising through the frozen downstream layers. Across 14 benchmarks and four judges from three model families, HRS outperforms all evaluated frozen-weight baselines. With Qwen2.5-7B, it achieves the lowest Jensen-Shannon distance on 13 of 14 benchmarks, reducing the mean from 0.565 to 0.352 versus 0.377 for the strongest calibrator. HRS also raises mean Spearman correlation with aggregate human scores from 0.449 to 0.579, improves population-level estimation under limited annotation budgets, and trains about 88 times fewer parameters than distributional fine-tuning at a modest cost in alignment.
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Post and Telecommunications(北京邮电大学)
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机学院,北京大学);StepFun;Xi’an Jiaotong University(西安交通大学)
Beyond the Surface: Measuring Self-Preference in LLM Judgments
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学首都人工智能学院);Huawei Poisson Lab(华为Poisson实验室);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理重点实验室)
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Arizona State University(亚利桑那州立大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Northwestern University(西北大学);University of California, Berkeley(加州大学伯克利分校);Emory University(埃默里大学)
Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
Technische Universität Berlin(柏林技术大学);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究院);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Fraunhofer Heinrich Hertz Institute(弗劳恩霍夫海因里希·赫茨研究所);Marburg University(马尔堡大学);Centre for European Research in Trusted AI (CERTAIN)(欧洲可信人工智能研究中心)
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU)HiTZ中心);University of Alicante (UA)(阿利坎特大学(UA));GOI Institute, Basque Summer University (UEU)(巴斯克夏季大学(UEU)GOI研究所)
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation
School of Computer Science and Engineering, The Hebrew University of Jerusalem(计算机科学与工程学院,特拉维夫大学);IBM Research(IBM研究院);The Allen Institute for AI (AI2)(人工智能研究所)
Improve LLM-as-a-Judge Ability as a General Ability
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);Institute of Artificial Intelligence, China Telecom (TeleAI)(中国电信人工智能研究院(TeleAI));Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Tasks
Faculty of Computer Science(计算机科学系);UniVie Doctoral School Computer Science(维也纳大学计算机科学博士学院);Faculty of Philological and Cultural Studies(文学与文化研究系);University of Vienna(维也纳大学);Saarland University(萨尔兰大学);Eastern Institute of Technology(东部技术研究所);Department of Statistics(统计系);LMU Munich(慕尼黑莱茵河大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)
Do Large Language Models Judge Error Severity Like Humans?
School of Psychology(心理学系);Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning(湖北省人工智能与智能学习重点实验室);National Language Resources Monitor and Research Center for Network Media(网络媒体语言资源监测与研究中心);School of Computer Science(计算机科学学院)
Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge
City University of Hong Kong(香港城市大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));McGill University(麦吉尔大学);MILA
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)(国立信息学研究所);University of Amsterdam(阿姆斯特丹大学);University of Tokyo(东京大学);Hitotsubashi University(一桥大学);Centrum Wiskunde & Informatica (CWI)(荷兰数学与计算机科学研究中心);Utrecht University(乌得勒支大学)
Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
Department of Computer Science and Information Engineering(计算机科学与信息工程系);National Taiwan University(国立台湾大学);Institute of Information Science, Academia Sinica(学术院信息研究所);AI Research Center (AINTU)(人工智能研究中心)