BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
Southern University of Science and Technology（南方科技大学）；Alibaba Group（阿里巴巴集团）；Beihang University（北航）；Shanghai University of Finance and Economics（上海财经大学）
LLM-as-a-Judge has been widely adopted across various research and practical applications, yet the robustness and reliability of its evaluation remain a critical issue. A core challenge it faces is bias, which has primarily been studied in terms of known biases and their impact on evaluation outcomes, while automated and systematic exploration of potential unknown biases is still lacking. Nevertheless, such exploration is crucial for enhancing the robustness and reliability of evaluations. To bridge this gap, we propose BiasScope, a LLM-driven framework for automatically and at scale discovering potential biases that may arise during model evaluation. BiasScope can uncover potential biases across different model families and scales, with its generality and effectiveness validated on the JudgeBench dataset. It overcomes the limitations of existing approaches, transforming bias discovery from a passive process relying on manual effort and predefined bias lists into an active and comprehensive automated exploration. Moreover, based on BiasScope, we propose JudgeBench-Pro, an extended version of JudgeBench and a more challenging benchmark for evaluating the robustness of LLM-as-a-judge. Strikingly, even powerful LLMs as evaluators show error rates above 50\% on JudgeBench-Pro, underscoring the urgent need to strengthen evaluation robustness and to mitigate potential biases further.
An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
Singapore Management University（新加坡国立管理学院）；DGIST（韩国高等智能科学研究院）；Monash University（墨尔本大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Huawei Software Engineering Application Technology Lab（华为软件工程应用技术实验室）；Zhejiang University（浙江大学）
Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
East China Normal University（华东师范大学）；Shanghai Institute of Artificial Intelligence for Education（上海人工智能教育研究院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Yangzhou University（扬州大学）；School of Education and Intelligent Education Research Center, Yangzhou University（扬州大学教育学院与智能教育研究中心）；School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）
Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants
Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；BMW Group（宝马集团）；Technical University of Munich（慕尼黑工业大学）；RWTH Aachen University（亚琛工业大学）；fortiss GmbH（fortiss有限公司）
Quantifying and Mitigating Self-Preference Bias of LLM Judges
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China（复杂实验室、计算机科学与工程学院、电子科技大学、成都、中国）
TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Institute of Science Tokyo（东京科学研究院）；Nanjing University（南京大学）；Westlake University（西湖大学）；Southeast University（东南大学）
Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
Department of Artificial Intelligence, Chung-Ang University, Seoul, Republic of Korea（Chung-Ang 大学人工智能系）；Department of Industrial Engineering, Seoul National University, Seoul, Republic of Korea（首尔国立大学工业工程系）
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)（国立信息学研究所）；University of Amsterdam（阿姆斯特丹大学）；University of Tokyo（东京大学）；Hitotsubashi University（一桥大学）；Centrum Wiskunde & Informatica (CWI)（荷兰数学与计算机科学研究中心）；Utrecht University（乌得勒支大学）
Florida International University（佛罗里达国际大学）；Meta Reality Labs（Meta现实实验室）
Large Language Models (LLMs) have demonstrated remarkable progress through preference-based fine-tuning, which critically depends on the quality of the underlying training data. While human feedback is essential for improving data quality, it is costly and does not scale well. In this paper, we introduce Refine-n-Judge, an automated iterative approach that leverages a single LLM as both a refiner and a judge to enhance dataset quality. Unlike existing iterative refinement methods, Refine-n-Judge employs an LLM to both generate refinements and explicitly evaluate each improvement, ensuring that…
Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation
Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Allen Institute for AI（艾伦人工智能研究所）；Microsoft（微软）；University of Arizona（亚利桑那大学）；University of Washington（华盛顿大学）
