Large language models (LLMs) are increasingly used as automated judges to evaluate recommendation systems, search engines, and other subjective tasks, where relying on human evaluators can be costly, time-consuming, and unscalable. LLMs offer an efficient solution for continuous, automated evaluation. However, since the systems that are built and improved with these judgments are ultimately designed for human use, it is crucial that LLM judgments align closely with human evaluators to ensure such systems remain human-centered. On the other hand, aligning LLM judgments with human evaluators is challenging due to individual variability and biases in human judgments. We propose a simple yet effective framework to align LLM judgments with individual human evaluators or their aggregated judgments, without retraining or fine-tuning the LLM. Our approach learns a linear mapping between the LLM's outputs and human judgments, achieving over 142% average improvement in agreement across 29 tasks with only a small number of calibration examples used for training. Notably, our method works in zero-shot and few-shot settings, exceeds inter-human agreement on four out of six tasks, and enables smaller LLMs to achieve performance comparable to that of larger models.
Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院，中国沈阳）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系，中国北京）；Software College, Northeastern University, Shenyang, China（东北大学软件学院，中国沈阳）；Huawei, China（华为，中国）
From Prediction to Justification: Aligning Sentiment Reasoning with Human Rationale via Reinforcement Learning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Shanghai Qiji Zhifeng Co., Ltd.（上海启智丰科技有限公司）；Ocean University of China（中国海洋大学）
Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis
School of Computer Science and Technology, Dalian University of Technology, China（大连理工大学计算机科学与技术学院）；Tencent（腾讯）；Peking University（北京大学）；Tongji University（同济大学）
