In this paper, we present a mathematical model to capture various factors which may influence the accuracy of a competitive group recommendation system. We apply this model to peer review systems, i.e., conference or research grants review, which is an essential component in our scientific community. We explore number of important questions, i.e., how will the number of reviews per paper affect the accuracy of the overall recommendation? Will the score aggregation policy influence the final recommendation? How reviewers' preference may affect the accuracy of the final recommendation? To answer these important questions, we formally analyze our model. Through this analysis, we obtain the insight on how to design a randomized algorithm which is both computationally efficient and asymptotically accurate in evaluating the accuracy of a competitive group recommendation system. We obtain number of interesting observations: i.e., for a medium tier conference, three reviews per paper is sufficient for a high accuracy recommendation. For prestigious conferences, one may need at least seven reviews per paper to achieve high accuracy. We also propose a heterogeneous review strategy which requires equal or less reviewing workload, but can improve over a homogeneous review strategy in recommendation accuracy by as much as 30% . We believe our models and methodology are important building blocks to study competitive group recommendation systems.
Traditional statistical representations outperform generative AI in identifying expert peer reviewers
Michigan State University(密歇根州立大学);Masaryk University(马萨里克大学);Max Planck Institute for Astrophysics(马克斯·普朗克天体物理研究所);Joint ALMA Observatory(ALMA联合天文台);Stanford University(斯坦福大学);European Southern Observatory(欧洲南方天文台);Space Telescope Science Institute(太空望远镜科学研究所)