It is not easy to rationalize how peer review, as the current grassroots of science, can work based on voluntary contributions of reviewers. There is no rationale to write impartial and thorough evaluations. Consequently, there is no risk in submitting low-quality work by authors. As a result, scientists face a social dilemma: if everyone acts according to his or her own self-interest, low scientific quality is produced. Still, in practice, reviewers as well as authors invest high effort in reviews and submissions.
We examine how the increased relevance of public good benefits (journal impact factor), the editorial policy of handling incoming reviews, and the acceptance decisions that take into account reputational information can help the evolution of high-quality contributions from authors. High effort from the side of reviewers is problematic even if authors cooperate: reviewers are still best off by producing low-quality reviews, which does not hinder scientific development, just adds random noise and unnecessary costs to it. We show with agent-based simulations that tacit agreements between authors that are based on reciprocity might decrease these costs, but does not result in superior scientific quality. Our study underlines why certain self-emerged current practices, such as the increased importance of journal metrics, the reputation-based selection of reviewers, and the reputation bias in acceptance work efficiently for scientific development. Our results find no answers, however, how the system of peer review with impartial and thorough evaluations could be sustainable jointly with rapid scientific development.
Peer review is an integral component of contemporary science. While peer review focuses attention on promising and interesting science, it also encourages scientists to pursue some questions at the expense of others. Here, we use ideas from forecasting assessment to examine how two modes of peer review -- ex ante review of proposals for future work and ex post review of completed science -- motivate scientists to favor some questions instead of others. Our main result is that ex ante and ex post peer review push investigators toward distinct sets of scientific questions. This tension arises be…
We study how publicity, in the form of highlighting in the science press, affects the citations of research papers. After a brief review of prior work, we analyze papers published in Physical Review Letters (PRL) that are highlighted across eight different platforms. Using multiple linear regression we identify how each platform contributes to citations. We also analyze how frequently the highlighted papers end up in the top 1% cited papers in their field. We find that the strongest predictors of medium-term citation impact -- up to 7 years post-publication -- are Viewpoints in Physics, follow…
Demanding peer review is associated with higher impact in published science
School of System Science, Beijing Normal University（北京师范大学系统科学学院）；School of Public Policy and Management, University of Chinese Academy of Sciences（中国科学院大学公共政策与管理学院）；Business School, Beijing Normal University（北京师范大学商学院）
US funding agencies alone distribute a yearly total of roughly $65B dollars largely through the process of proposal peer review: scientists compete for project funding by submitting grant proposals which are evaluated by selected panels of peer reviewers. Similar funding systems are in place in most advanced democracies. However, in spite of its venerable history, proposal peer review is increasingly struggling to deal with the increasing mismatch between demand and supply of research funding.
The paper describes a potential platform to facilitate academic peer review with emphasis on early-stage research. This platform aims to make peer review more accurate and timely by rewarding reviewers on the basis of peer prediction algorithms. The algorithm uses a variation of Peer Truth Serum for Crowdsourcing (Radanovic et al., 2016) with human raters competing against a machine learning benchmark. We explain how our approach addresses two large productive inefficiencies in science: mismatch between research questions and publication bias. Better peer review for early research creates addi…
This paper appraises the concordance between bibliometrics and peer review, by drawing evidence from the data of two experiments realized by the Italian governmental agency for research evaluation. The experiments were performed for validating the dual system of evaluation, consisting in the interchangeable use of bibliometyrics and peer review, adopted by the agency in the research assessment exercises. The two experiments were based on stratified random samples of journal articles. Each article was scored by bibliometrics and by peer review. The degree of concordance between the two evaluati…
Institute for Biocomputation and Physics of Complex Systems, University of Zaragoza（萨拉戈萨大学复杂系统生物计算与物理研究所）；Department of Theoretical Physics, Faculty of Sciences, University of Zaragoza（萨拉戈萨大学理学院理论物理系）
While ancient scientists often had patrons to fund their work, peer review of proposals for the allocation of resources is a foundation of modern science. A very common method is that proposals are evaluated by a small panel of experts (due to logistics and funding limitations) nominated by the grant-giving institutions. The expert panel process introduces several issues - most notably: 1) biases introduced in the selection of the panel. 2) experts have to read a very large number of proposals. Distributed Peer Review promises to alleviate several of the described problems by distributing the …
