Existing large language models (LLMs) evaluation methods typically focus on testing the performance on some closed-environment and domain-specific benchmarks with human annotations. In this paper, we explore a novel unsupervised evaluation direction, utilizing peer-review mechanisms to measure LLMs automatically. In this setting, both open-source and closed-source LLMs lie in the same environment, capable of answering unlabeled questions and evaluating each other, where each LLM's response score is jointly determined by other anonymous ones. To obtain the ability hierarchy among these models, we assign each LLM a learnable capability parameter to adjust the final ranking. We formalize it as a constrained optimization problem, intending to maximize the consistency of each LLM's capabilities and scores. The key assumption behind is that high-level LLM can evaluate others' answers more accurately than low-level ones, while higher-level LLM can also achieve higher response scores. Moreover, we propose three metrics called PEN, CIN, and LIS to evaluate the gap in aligning human rankings. We perform experiments on multiple datasets with these metrics, validating the effectiveness of the proposed approach.
Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review
Shanghai Jiao Tong University(上海交通大学);Georgia Institute of Technology(佐治亚理工学院);Shanghai AI Laboratory(上海人工智能实验室);University of Georgia(佐治亚大学);Oxford University(牛津大学)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔高等商学院);ServiceNow Research(ServiceNow研究);Université de Montréal(蒙特利尔大学);Canada CIFAR Chair(加拿大 CIFAR 席位);Polytechnique Montréal(蒙特利尔理工学院)
本文提出 ReviewerToo 框架,用于研究 AI 辅助同行评审,在 ICLR 2025 数据集上验证其性能接近人类平均水平,并指出 AI 的优势与局限,提出集成指南,以提升评审的一致性和公平性。
Peering Inside the Black Box: Uncovering LLM Errors in Optimization Modelling through Component-Level Evaluation
Computer Science Department, KFUPM, Dhahran 31261, Saudi Arabia;SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum \& Minerals, Dhahran, 31261, Saudi Arabia
SC-Taxo: Hierarchical Taxonomy Generation under Semantic Consistency Constraints using Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology Beijing(北京科技大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback
UKP Lab, TU Darmstadt and Hessian Center for AI (hessian.AI)(图林人工智能实验室、德累斯顿理工大学和黑森人工智能中心);MBZUAI(马克斯·普朗克人工智能研究所);The Allen Institute for AI (AI2)(人工智能研究院)
FMMD: A multimodal multidisciplinary dataset of open peer reviews from F1000Research
School of Computer Science and Engineering, Guangzhou Institute of Science and Technology(计算机科学与工程学院,广州科学与技术研究所);College of Foreign Languages and Cultures, Xiamen University(外语学院,厦门大学);Science and Education Evaluation Lab, Guangzhou Institute of Science and Technology(科学与教育评估实验室,广州科学与技术研究所)
Benchmark^2: Systematic Evaluation of LLM Benchmarks
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Washington University in St. Louis(华盛顿大学圣路易斯分校);Xiaohongshu Inc.(小红书公司)