We propose the PeerRank method for peer assessment. This constructs a grade for an agent based on the grades proposed by the agents evaluating the agent. Since the grade of an agent is a measure of their ability to grade correctly, the PeerRank method weights grades by the grades of the grading agent. The PeerRank method also provides an incentive for agents to grade correctly. As the grades of an agent depend on the grades of the grading agents, and as these grades themselves depend on the grades of other agents, we define the PeerRank method by a fixed point equation similar to the PageRank method for ranking web-pages. We identify some formal properties of the PeerRank method (for example, it satisfies axioms of unanimity, no dummy, no discrimination and symmetry), discuss some examples, compare with related work and evaluate the performance on some synthetic data. Our results show considerable promise, reducing the error in grade predictions by a factor of 2 or more in many cases over the natural baseline of averaging peer grades.
AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment
Department of Computer Science, Sapienza University of Rome（计算机科学系，罗马萨皮恩扎大学）；Department of Computer, Control, and Management Engineering, Sapienza University of Rome（计算机、控制与管理工程系，罗马萨皮恩扎大学）；eZecute S.R.L.（eZecute公司）
My part is bigger than yours -- assessment within a group of peers
AGH University of Krakow, WEAIiIB, Applied Computer Science Department（克拉科夫AGH大学，WEAIiIB，应用计算机科学系）；AGH University of Krakow, Faculty of Applied Mathematics（克拉科夫AGH大学，应用数学学院）
Paper Quality Assessment based on Individual Wisdom Metrics from Open Peer Review
Department of Electrical Engineering and Computer Science, MIT（麻省理工学院电气工程与计算机科学系）；Département de Psychologie, Université de Montréal, Canada（蒙特利尔大学心理学系）；Department of Experimental Psychology, University of Oxford, United Kingdom（牛津大学实验心理学系）；Department of Brain and Cognitive Sciences & McGovern Institute, MIT（麻省理工学院脑科学与认知科学系）
NovBench: Evaluating Large Language Models on Academic Paper Novelty Assessment
School of Economics and Management, Nanjing University of Science and Technology（南京理工大学经济管理学院）；School of Electronic Engineering and Computer Science, Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；School of Management, Anhui University（安徽大学管理学院）
