LLM-as-a-Judge has been widely applied to evaluate and compare different LLM alignmnet approaches (e.g., RLHF and DPO). However, concerns regarding its reliability have emerged, due to LLM judges' biases and inconsistent decision-making. Previous research has developed evaluation frameworks to assess reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address LLM internal inconsistency. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-Judge methods, leading to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM-as-a-Judge on alignment tasks by defining more theoretically interpretable evaluation metrics and explicitly mitigating LLM internal inconsistency from reliability metrics. We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges, which facilitates practitioners to choose LLM judges for alignment tasks. In the experiments, we examine effects of diverse prompt templates on LLM-judge reliability and also demonstrate our developed framework by comparing various LLM judges on two common alignment datasets (i.e., TL;DR Summarization and HH-RLHF-Helpfulness). Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.
Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judge
Business School, Shandong University of Technology（山东理工大学商学院）；Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Guangdong Institute of Intelligent Science and Technology（广东智能科学与技术研究院）；Macau Millennium College（澳门 millennium 学院）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）
From the One, Judge of the Whole: Typed Entailment Graph Construction with Predicate Generation
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Center for Data Science, Peking University（北京大学数据科学中心）；The MOE Key Laboratory of Computational Linguistics, Peking University（北京大学计算语言学教育部重点实验室）
Self-Exploring Language Models for Explainable Link Forecasting on Temporal Graphs via Reinforcement Learning
University of Cambridge（剑桥大学）；Mila - Quebec AI Institute（魁北克人工智能研究院）；McGill University（麦吉尔大学）；University of Oxford（牛津大学）；National University of Singapore（新加坡国立大学）；University of Stuttgart（斯图加特大学）；Amazon（亚马逊公司）；AITHYRA
CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China（上海交通大学计算机科学学院）；Shanghai KEY Laboratory of Integrated Administration Technologies for Information Security, Shanghai, China（上海信息安全集成管理技术重点实验室）；School of Politics, National Defense University, Shanghai, China（国防大学政治学院）
CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement
Business School, Shandong University of Technology, Shandong, China（山东科技大学商学院）；Institute of International Language Services Studies, Macau Millennium College, Macau SAR, China（国际语言服务研究所，澳门 Millennium 学院）
Rethinking Creativity Evaluation: A Critical Analysis of Existing Creativity Evaluations
Graduate Institute of Communication Engineering, National Taiwan University（台湾大学通信工程研究院）；Computer Science Department, University of California, Los Angeles（加州大学洛杉矶分校计算机科学系）；Graduate Institute of Networking and Multimedia, National Taiwan University（台湾大学网络与多媒体研究院）；Department of Electrical Engineering, National Taiwan University（台湾大学电子工程系）
HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）；Pretraining Team, xAI（预训练团队，xAI）；Hong Kong University of Science and Technology (GuangZhou)（香港科技大学（广州））
Dementia Through Different Eyes: Explainable Modeling of Human and LLM Perceptions for Early Awareness
Faculty of Data and Decision Sciences, Technion（技术学院数据与决策科学学院）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
Cognition Chain for Explainable Psychological Stress Detection on Social Media
University of Oxford（牛津大学）；Tsinghua University（清华大学）；Beijing Normal University（北京师范大学）；Wuhan University（武汉大学）；Oxford Suzhou Centre for Advanced Research（牛津苏州高等研究院）
University of Michigan（密歇根大学）；University Pompeu Fabra（庞培法布拉大学）；Federal University of Minas Gerais（米纳斯吉拉斯联邦大学）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；MIT（麻省理工学院）
