Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can age and enter model training corpora, while centralized judges introduce their own scoring preferences.
We cast LLM evaluation as a multi-agent peer-assessment problem and introduce Agents-as-Peers, a distributed evaluation protocol in which every agent generates tasks, answers the shared task set, and judges the answers of its peers, with no privileged evaluator.
Self-ratings are excluded from the ranking, and judging is repeated under controlled identity and presentation regimes so that self bias, identity effects, presentation-position sensitivity, and judge severity are measured rather than left implicit.
In an instantiation with 12 LLMs and 420 agent-generated questions, Agents-as-Peers aggregates 55,440 non-self peer judgments per regime.
Most agents assign their own answers higher scores than those answers receive from peer judges; visible identity changes scores for some agents; the first fixed answer slot is associated with scores 0.39 points higher than the shuffled baseline; and judges differ markedly in severity.
On labeled benchmarks, peer scores align strongly with accuracy on TruthfulQA (Pearson r=0.904) and GSM8K (r=0.873). On TruthfulQA, the peer-score correlation is substantially higher than the self-score correlation (r=0.538).
A benchmark-overlap screen and item-level human audit further characterize the endogenous task set.
These results support Agents-as-Peers as a cohort-relative measurement protocol whose ranking is reported together with the biases and rater effects that shape it.
Code and dataset: https://github.com/caura-ai/caura-PeerRank
AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
Department of Computer Science, Sapienza University of Rome（计算机科学系，罗马萨皮恩扎大学）；Department of Computer, Control, and Management Engineering, Sapienza University of Rome（计算机、控制与管理工程系，罗马萨皮恩扎大学）；eZecute S.R.L.（eZecute公司）
My part is bigger than yours -- assessment within a group of peers
AGH University of Krakow, WEAIiIB, Applied Computer Science Department（克拉科夫AGH大学，WEAIiIB，应用计算机科学系）；AGH University of Krakow, Faculty of Applied Mathematics（克拉科夫AGH大学，应用数学学院）
Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review
Department of Electrical Engineering and Computer Science, MIT（麻省理工学院电气工程与计算机科学系）；Département de Psychologie, Université de Montréal, Canada（蒙特利尔大学心理学系）；Department of Experimental Psychology, University of Oxford, United Kingdom（牛津大学实验心理学系）；Department of Brain and Cognitive Sciences & McGovern Institute, MIT（麻省理工学院脑科学与认知科学系）
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
School of Economics and Management, Nanjing University of Science and Technology（南京理工大学经济管理学院）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；School of Management, Anhui University（安徽大学管理学院）
