Split and Merge: Aligning Position Biases in LLM-based Evaluators
Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology（哈尔滨工业大学）；Nanyang Technological University（南洋理工大学）
Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, Yang Liu
备注：
Accepted by EMNLP 2024. Please cite the conference version of this paper, e.g., "Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2024. Split and Merge: Aligning Position Biases in LLM-based Evaluators. (EMNLP 2024)"
Large language models (LLMs) have shown promise as automated evaluators for assessing the quality of answers generated by AI systems. However, these LLM-based evaluators exhibit position bias, or inconsistency, when used to evaluate candidate answers in pairwise comparisons, favoring either the first or second answer regardless of content. To address this limitation, we propose PORTIA, an alignment-based system designed to mimic human comparison strategies to calibrate position bias in a lightweight yet effective manner. Specifically, PORTIA splits the answers into multiple segments, aligns similar content across candidate answers, and then merges them back into a single prompt for evaluation by LLMs. We conducted extensive experiments with six diverse LLMs to evaluate 11,520 answer pairs. Our results show that PORTIA markedly enhances the consistency rates for all the models and comparison forms tested, achieving an average relative improvement of 47.46%. Remarkably, PORTIA enables less advanced GPT models to achieve 88% agreement with the state-of-the-art GPT-4 model at just 10% of the cost. Furthermore, it rectifies around 80% of the position bias instances within the GPT-4 model, elevating its consistency rate up to 98%. Subsequent human evaluations indicate that the PORTIA-enhanced GPT-3.5 model can even surpass the standalone GPT-4 in terms of alignment with human evaluators. These findings highlight PORTIA's ability to correct position bias, improve LLM consistency, and boost performance while keeping cost-efficiency. This represents a valuable step toward a more reliable and scalable use of LLMs for automated evaluations across diverse applications.
ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models
East China Normal University（华东师范大学）；Hasso Plattner Institute/University of Potsdam（哈索·普拉特纳研究所/波茨坦大学）；East China University of Science and Technology（华东理工大学）
Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge
National Institute of Informatics (NII)（国立信息学研究所）；University of Amsterdam（阿姆斯特丹大学）；University of Tokyo（东京大学）；Hitotsubashi University（一桥大学）；Centrum Wiskunde & Informatica (CWI)（荷兰数学与计算机科学研究中心）；Utrecht University（乌得勒支大学）
VALUEFLOW: Toward Pluralistic and Steerable Value-based Alignment in Large Language Models
Department of Electrical and Computer Engineering, Seoul National University（首尔国立大学电气与计算机工程系）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔国立大学人工智能交叉学科项目）
Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling
School of Computer Science and Engineering, Northeastern University, Shenyang, China（东北大学计算机科学与工程学院，中国沈阳）；Department of Computer Science and Technology, Tsinghua University, Beijing, China（清华大学计算机科学与技术系，中国北京）；Software College, Northeastern University, Shenyang, China（东北大学软件学院，中国沈阳）；Huawei, China（华为，中国）
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
Vellore Institute of Technology（维洛雷理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）；Northwestern University（西北大学）；Yale University（耶鲁大学）；Algoverse AI Research（Algoverse AI研究）
