Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
Beihang University, Beijing, China(北京航空航天大学);Zhongguancun Laboratory, Beijing, China(中关村实验室);Beijing University of Posts and Telecommunications(北京邮电大学);Hong Kong University of Science and Technology(香港科学与技术大学);Xi'an Jiaotong University, Xi'an, China(西安交通大学);Tsinghua University, Beijing, China(清华大学);Singapore University of Technology and Design(新加坡科技设计大学)
We propose LLM-PeerReview, an unsupervised LLM Ensemble method that selects the most ideal response from multiple LLM-generated candidates for each query, harnessing the collective wisdom of multiple models with diverse strengths. LLM-PeerReview is built on a novel, peer-review-inspired framework that offers a transparent and interpretable mechanism, while remaining fully unsupervised for flexible adaptability and generalization. Specifically, it operates in three stages: For scoring, we use the emerging LLM-as-a-Judge technique to evaluate each response by reusing multiple LLMs at hand; For reasoning, we can apply a straightforward averaging strategy or a principled graphical model-based truth inference algorithm to aggregate multiple scores to produce a final score for each response; Finally, the highest-scoring response is selected as the best ensemble output. LLM-PeerReview is conceptually simple and empirically powerful. Our results across four datasets show that the two variants of the proposed approach outperform the advanced model Smoothie-Global by 6.9% and 7.3% points, cross diverse task types including factual recall QA, math reasoning, and instruction following.
Notably, we also establish a carefully curated benchmark suite for LLM Ensemble, integrating 12 methods across four classic datasets and three task families, all evaluated under a rigorous and consistent protocol. We hope this repository will help researchers reproduce the LLM Ensemble baselines.
Can We Triage LLM Translation Errors in Classical Texts Without Human References? Source Novelty, GEMBA Scoring, and Budgeted Review through Pali-to-English Translation
Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt(普遍知识处理实验室(UKP实验室)、计算机科学系和海斯曼人工智能中心(hessian.AI)、达姆施塔特技术大学);Department of Computer Science at Queens College, City University of New York (CUNY)(纽约城市大学皇后学院计算机科学系)
PPAI: Enabling Personalized LLM Agent Interoperability for Collaborative Edge Intelligence
Hong Kong RGC General Research Fund(香港研究资助局一般研究基金);Research Impact Fund(研究影响基金);Collaborative Research Fund(协作研究基金);NSFC/RGC Collaborative Research Scheme(国家自然科学基金/香港研究资助局协作研究计划);Areas of Excellence Scheme(卓越领域计划);InnoHK (HKGAI)(创新科技署(HKGAI))
Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts
University of California, Los Angeles(加州大学洛杉矶分校);Sea AI Lab(Sea AI 实验室);Stanford University(斯坦福大学);University of Oxford(牛津大学);Yale University(耶鲁大学);NTU(南洋理工大学);NUS(新加坡国立大学);Boston University(波士顿大学)
Making AI-Assisted Grant Evaluation Auditable without Exposing the Model
Informatics Institute, Istanbul Technical University, Istanbul, Türkiye(伊斯坦布尔技术大学信息学院,伊斯坦布尔,土耳其);Securify Information Technology and Security Training Consulting Inc., Ankara, Türkiye(Securify信息科技与安全培训咨询公司,安卡拉,土耳其)
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Singapore Management University(新加坡管理大学);Tsinghua University(清华大学);University of California, Los Angeles(加州大学洛杉矶分校);Beijing University of Posts and Telecommunications(北京邮电大学)
Auditing medical multi-agent AI reveals risks of false consensus
National Engineering Research Center for Software Engineering, Peking University(北京大学软件工程国家工程研究中心);School of Computing and Data Science, The University of Hong Kong(香港大学计算机与数据科学学院);Department of Nephrology, Peking University Third Hospital(北京大学第三医院肾内科);Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Department of Lymphoma, Peking University Cancer Hospital & Institute(教育部癌症发生与转化研究重点实验室、北京大学肿瘤医院淋巴瘤科);Department of Automation, Tsinghua University(清华大学自动化系);Centre for Medical Informatics, The University of Edinburgh(爱丁堡大学医学信息学中心);Health Data Research UK(英国健康数据研究机构);Lee Kong Chian School of Medicine, Nanyang Technological University(南洋理工大学李科贤医学院);State Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University(南京大学新型软件技术国家重点实验室、计算机科学学院)
LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
University of South Florida(佛罗里达南大学);Missouri University of Science and Technology(密苏里科技大学);University of Alabama(阿拉巴马大学);Florida International University(佛罗里达国际大学);University of Cincinnati(辛辛那提大学);George Mason University(乔治·梅森大学)