When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found. However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review. We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review. In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF. The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review. To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.
Comments received from referees during peer-review were analysed to determine the rates of duplication and partial duplication. It is very unusual for 2 different referees to submit identical comments, so the rare cases where this happens are of interest. In some cases, it appears that paper-mills create fake referee accounts and use them to submit fake peer-review reports. These include comments that are copied and pasted across multiple reviews. Searching for duplication in referee comments is therefore an effective method to search for misconduct generally, since the forms of misconduct com…
Since a number of journals specifically focus on the review and publication of data sets, reviewing their policies seems an appropriate place to start in assessing what existing practice looks like in the 'real world' of reviewing and publishing data. This article outlines a study of the publicly available peer review policies of 39 scientific publications that publish data papers to discern which criteria are most and least frequently referenced. It also compares current practice with proposed criteria published in 2012.
The use of bibliometric indicators would simplify research assessments. The 2014 Research Excellence Framework (REF) is a peer review assessment of UK universities, whose results can be taken as benchmarks for bibliometric indicators. In this study we use the REF results to investigate whether the ep index and a top percentile of most cited papers could substitute for peer review. The probability that a random university's paper reaches a certain top percentile in the global distribution of papers is a power of the ep index, which can be calculated from the citation-based distribution of unive…
In computer science, conferences and journals conduct peer review in order to decide what to publish. Many have pointed out the inherent weaknesses in peer review, including those of bias, quality, and accountability. Many have suggested and adopted refinements of peer review, for instance, double blind peer review with author rebuttals. In this essay, I argue that peer review as currently practiced conflates the sensible idea of getting comments on a paper with the irrevocably-flawed one that we either accept or reject the paper, which I term gatekeeping. If we look at the two separately, the…
The importance of peer-review in the scientific process can not be overestimated. Yet, due to increasing pressures of research and exponentially growing number of publications the task faced by the referees becomes ever more difficult. We discuss here a few possible improvements that would enable more efficient review of the scientific literature, using the growing Internet connectivity. In particular, a practical automated model for providing the referees with references to papers that might have strong relationship with the work under review, based on general network properties of citations …
In this paper peer review reliability is investigated based on peer ratings of research teams at two Belgian universities. It is found that outcomes can be substantially influenced by the different ways in which experts attribute ratings. To increase reliability of peer ratings, procedures creating a uniform reference level should be envisaged. One should at least check for signs of low reliability, which can be obtained from an analysis of the outcomes of the peer evaluation itself. The peer review results are compared to outcomes from a citation analysis of publications by the same teams, in…
The advancement of various fields of science depends on the actions of individual scientists via the peer review process. The referees' work patterns and stochastic nature of decision making both relate to the particular features of refereeing and to the universal aspects of human behavior. Here, we show that the time a referee takes to write a report on a scientific manuscript depends on the final verdict. The data is compared to a model, where the review takes place in an ongoing competition of completing an important composite task with a large number of concurrent ones - a Deadline -effect…
Bibliometric analysis has firmly conquered its place as an instrument for evaluation and international comparison of performance levels. Consequently, differences in coverage by standard bibliometric databases installed a dichotomy between on the one hand the well covered 'exact' sciences, and on the other hand most of the social sciences and humanities with a more limited coverage (Nederhof, 2006). Also the latter domains need to be able to soundly demonstrate their level of performance and claim or legitimate funding accordingly. An important part of the output volume in social sciences appe…
Criteria are an essential component of any procedure for assessing merit. Yet, little is known about the criteria peers use in assessing grant applications. In this systematic review we therefore identify and synthesize studies that examine grant peer review criteria in an empirical and inductive manner. To facilitate the synthesis, we introduce a framework that classifies what is generally referred to as 'criterion' into an evaluated entity (i.e. the object of evaluation) and an evaluation criterion (i.e. the dimension along which an entity is evaluated). In total, this synthesis includes 12 …
This paper describes the method for ex-post peer review evaluation per research discipline used at the Vrije Universiteit Brussel (VUB) and summarizes the outcomes obtained from it. The method produces pertinent advice and triggers responses - at the level of the individual researcher, the research team and the university's research management - for the benefit of research quality, competitivity and visibility. Imposed reflection and contacts during and after the evaluation procedure modify the individual researcher's attitude, improve the research teams' strategies and allow for the extractio…
The peer-review process is the most widely accepted certification mechanism for officially accepting the written results of researchers within the scientific community. An essential component of peer-review is the identification of competent referees to review a submitted manuscript. This article presents an algorithm to automatically determine the most appropriate reviewers for a manuscript by way of a co-authorship network data structure and a relative-rank particle-swarm algorithm. This approach is novel in that it is not limited to a pre-selected set of referees, is computationally efficie…
A `peer-review system' in the context of judging research contributions, is one of the prime steps undertaken to ensure the quality of the submissions received, a significant portion of the publishing budget is spent towards successful completion of the peer-review by the publication houses. Nevertheless, the scientific community is largely reaching a consensus that peer-review system, although indispensable, is nonetheless flawed. A very pertinent question therefore is "could this system be improved?". In this paper, we attempt to present an answer to this question by considering a massive da…
Peer review aims to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts' content t…
Peer review is a cornerstone of modern scientific endeavor. However, there is growing consensus that several limitations of the current peer review system, from lack of incentives to reviewers to lack of transparency, risks to undermine its benefits. Here, we introduce the PRINCIPIA (http://www.principia.network/) framework for peer-review of scientific outputs (e.g., papers, grant proposals or patents). The framework allows key players of the scientific ecosystem -- including existing publishing groups -- to create and manage peer-reviewed journals, by building a free market for reviews and p…
Nanjing University of Science and Technology（南京理工大学）；Institute of Scientific and Technical Information of China（中国科学技术信息研究所）
Purpose The purpose of this paper is to explore which structures of academic articles referees would pay more attention to, what specific content referees focus on, and whether the distribution of PRC is related to the citations. Design/methodology/approach Firstly, utilizing the feature words of section title and hierarchical attention network model (HAN) to identify the academic article structures. Secondly, analyzing the distribution of PRC in different structures according to the position information extracted by rules in PRC. Thirdly, analyzing the distribution of feature words of PRC ext…
Pre-review to Peer review: Pitfalls of Automating Reviews using Large Language Models
AllSci Corp（AllSci公司）；Sunwater Capital（Sunwater资本）；Kellogg School of Management, Northwestern University（西北大学凯洛格管理学院）；Dept. of Computer Science, Northern Illinois University（北伊利诺伊大学计算机科学系）；Institute of Foundation Models, MBZUAI（基础模型研究所，MBZUAI）
Systematic evaluations of publicly funded research typically employ a combination of bibliometrics and peer review, but it is not known whether the bibliometric component introduces biases. This article compares three alternative mechanisms for scoring 73,612 UK Research Excellence Framework (REF) journal articles from all 34 field-based Units of Assessment (UoAs) 2014-17: peer review, field normalised citations, and journal average field normalised citation impact. All three were standardised into a four-point scale. The results suggest that in almost all academic fields, bibliometric scoring…
Commitment Checklist: Auditing Author Commitments in Peer Review
AIST, Japan（日本国家信息技术研究所）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany（图宾根大学计算机科学系通用知识处理实验室及应用网络安全国家研究中心ATHENE德国）
Peer review is the most common mechanism in place for assessing requests for resources in a large variety of scientific disciplines. One of the strongest criticisms to this paradigm is the limited reproducibility of the process, especially at largely oversubscribed facilities. In this and in a subsequent paper we address this specific aspect in a quantitative way, through a statistical study on proposal ranking at the European Southern Observatory. For this purpose we analysed a sample of about 15000 proposals, submitted by more than 3000 Principal Investigators over 8 years. The proposals wer…
Peer-review system has long been relied upon for bringing quality research to the notice of the scientific community and also preventing flawed research from entering into the literature. The need for the peer-review system has often been debated as in numerous cases it has failed in its task and in most of these cases editors and the reviewers were thought to be responsible for not being able to correctly judge the quality of the work. This raises a question "Can the peer-review system be improved?" Since editors and reviewers are the most important pillars of a reviewing system, we in this w…
Université de Montréal（蒙特利尔大学）；Université d’Ottawa（渥太华大学）
We analysed a dataset of scientific manuscripts that were submitted to various conferences in artificial intelligence. We performed a combination of semantic, lexical and psycholinguistic analyses of the full text of the manuscripts and compared them with the outcome of the peer review process. We found that accepted manuscripts scored lower than rejected manuscripts on two indicators of readability, and that they also used more scientific and artificial intelligence jargon. We also found that accepted manuscripts were written with words that are less frequent, that are acquired at an older ag…
During the Italian research assessment exercise, the national agency ANVUR performed an experiment to assess agreement between grades attributed to journal articles by informed peer review (IR) and by bibliometrics. A sample of articles was evaluated by using both methods and agreement was analyzed by weighted Cohen's kappas. ANVUR presented results as indicating an overall 'good' or 'more than adequate' agreement. This paper re-examines the experiment results according to the available statistical guidelines for interpreting kappa values, by showing that the degree of agreement, always in the…
