The impressive performance of large language models (LLMs) has attracted considerable attention from the academic and industrial communities. Besides how to construct and train LLMs, how to effectively evaluate and compare the capacity of LLMs has also been well recognized as an important yet difficult problem. Existing paradigms rely on either human annotators or model-based evaluators to evaluate the performance of LLMs on different tasks. However, these paradigms often suffer from high cost, low generalizability, and inherited biases in practice, which make them incapable of supporting the sustainable development of LLMs in long term. In order to address these issues, inspired by the peer review systems widely used in academic publication process, we propose a novel framework that can automatically evaluate LLMs through a peer-review process. Specifically, for the evaluation of a specific task, we first construct a small qualification exam to select "reviewers" from a couple of powerful LLMs. Then, to actually evaluate the "submissions" written by different candidate LLMs, i.e., the evaluatees, we use the reviewer LLMs to rate or compare the submissions. The final ranking of evaluatee LLMs is generated based on the results provided by all reviewers. We conducted extensive experiments on text summarization tasks with eleven LLMs including GPT-4. The results demonstrate the existence of biasness when evaluating using a single LLM. Also, our PRE model outperforms all the baselines, illustrating the effectiveness of the peer review mechanism.
AgentReview: Exploring Peer Review Dynamics with LLM Agents
Georgia Institute of Technology(佐治亚理工学院);University of Science and Technology of China(中国科学技术大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);University of California, Los Angeles(加州大学洛杉矶分校);William & Mary(威廉与玛丽学院)
What Can Natural Language Processing Do for Peer Review?
Technical University of Darmstadt(达姆施塔特工业大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Radboud University Nijmegen(奈梅亨拉德堡德大学);Carnegie Mellon University(卡内基梅隆大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);Bocconi University(博科尼大学);The University of Sydney(悉尼大学);Universität Hamburg(汉堡大学);University of British Columbia(不列颠哥伦比亚大学);University of Applied Sciences Darmstadt(达姆施塔特应用科学大学);Université Paris-Saclay(巴黎萨克雷大学);Monash University(莫纳什大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);University of Washington(华盛顿大学);Georgia Institute of Technology(佐治亚理工学院);IT University of Copenhagen(哥本哈根信息技术大学)
ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔高等商学院);ServiceNow Research(ServiceNow研究);Université de Montréal(蒙特利尔大学);Canada CIFAR Chair(加拿大 CIFAR 席位);Polytechnique Montréal(蒙特利尔理工学院)
本文提出 ReviewerToo 框架,用于研究 AI 辅助同行评审,在 ICLR 2025 数据集上验证其性能接近人类平均水平,并指出 AI 的优势与局限,提出集成指南,以提升评审的一致性和公平性。
CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)