Enabling Weak LLMs to Judge Response Reliability via Meta Ranking
Tsinghua University(清华大学);Weiyang College, Tsinghua University(清华大学未央书院);Institute of Intelligent Computing, Alibaba Group(阿里巴巴集团智能计算研究所);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能国际研究院)
Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Ji Zhang, Fei Huang, Yang Liu
Despite the strong performance of large language models (LLMs) across a wide range of tasks, they still have reliability issues. Previous studies indicate that strong LLMs like GPT-4-turbo excel in evaluating the reliability of responses from LLMs, but face efficiency and local deployment issues. Thus, to enable weak LLMs to effectively assess the reliability of LLM responses, we propose a novel cross-query-comparison-based method called $\textit{Meta Ranking}$ (MR). Unlike previous few-shot methods that solely based on in-context learning capabilities in LLMs, MR assesses reliability by pairwisely ranking the target query-response pair with multiple reference query-response pairs. We found that MR is highly effective in error detection for LLM responses, where weak LLMs, such as Phi-2, could surpass strong baselines like GPT-3.5-turbo, requiring only five reference samples and significantly improving efficiency. We further demonstrate that MR can enhance strong LLMs' performance in two practical applications: model cascading and instruction tuning. In model cascading, we combine open- and closed-source LLMs to achieve performance comparable to GPT-4-turbo with lower costs. In instruction tuning, we use MR for iterative training data filtering, significantly reducing data processing time and enabling LLaMA-7B and Phi-2 to surpass Alpaca-13B with fewer training tokens. These results underscore the high potential of MR in both efficiency and effectiveness.
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
Department of Computer Science, University of Exeter(埃克塞特大学计算机科学系);Institute of AI for Industries, Chinese Academy of Sciences(中国科学院工业人工智能研究所);Department of Mathematics and Computer Science, Eindhoven University of Technology(埃因霍温理工大学数学与计算机科学系)
GRASP: Deterministic argument ranking in interaction graphs
MPI-IS Tübingen(图宾根MPI研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(图宾根ELLIS研究所);Eberhard Karls Universität Tübingen(图宾根埃伯哈德·卡尔斯大学);LIONS, EPFL(EPFL的LIONS实验室)
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Trismik;Leverhulme Centre for the Future of Intelligence, University of Cambridge(Leverhulme智能未来研究中心,剑桥大学);Universitat Politècnica de València(瓦伦西亚理工大学);University of Cambridge(剑桥大学)
Can Argus Judge Them All? Comparing VLMs Across Domains
Bharati Vidyapeeth(巴哈提大学);Macquarie University(麦考瑞大学);DSEU-Okhla;Vivekananda Institute of Professional Studies(维维kananda专业研究学院);IIIT-Delhi(德里印度理工学院);Center for SDGC(SDGC中心);Stanford University(斯坦福大学)
PRISM: Calibrating Black-Box Judges with Counterfactual Probes
Massachusetts Institute of Technology(麻省理工学院);Sakana AI;Lab for Information and Decision Systems(信息与决策系统实验室);Statistics and Data Science Center(统计与数据科学中心);EECS(电气工程与计算机科学系)
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU)HiTZ中心);University of Alicante (UA)(阿利坎特大学(UA));GOI Institute, Basque Summer University (UEU)(巴斯克夏季大学(UEU)GOI研究所)
Benchmarking LLM Judges for Mobile Agent Evaluation
Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Toronto(多伦多大学);Shanghai University(上海大学);McMaster University(麦克马斯特大学)
Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Mathematical and Natural Sciences, Arizona State University(亚利桑那州立大学数学与自然科学学院)
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(信息处理国家重点实验室,计算机学院,北京大学);StepFun;Xi’an Jiaotong University(西安交通大学)
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
Arizona State University(亚利桑那州立大学);University of Illinois Chicago(伊利诺伊大学芝加哥分校);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);Northwestern University(西北大学);University of California, Berkeley(加州大学伯克利分校);Emory University(埃默里大学)
Meta-Judging with Large Language Models: Concepts, Methods, and Challenges
University of Coimbra, CISUC/LASI – Centre for Informatics and Systems of the University of Coimbra, Department of Informatics Engineering(科英布拉大学,CISUC/LASI——科英布拉大学信息与系统研究中心,信息工程系);Polytechnic University of Coimbra, Rua da Misericórdia, Lagar dos Cortiços, S. Martinho do Bispo, 3045-093 Coimbra, Portugal(科英布拉理工大学)