The peer review system as used in several computer science communities has several flaws including long review times, overloaded reviewers, as well as fostering of niche topics. These flaws decrease quality, lower impact, slowdown the innovation process, and lead to frustration of authors, readers, and reviewers. In order to fix this, we propose a new peer review system termed paper bricks. Paper bricks has several advantages over the existing system including shorter publications, better competition for new ideas, as well as an accelerated innovation process. Furthermore, paper bricks may be implemented with minimal change to the existing peer review systems.
Since a number of journals specifically focus on the review and publication of data sets, reviewing their policies seems an appropriate place to start in assessing what existing practice looks like in the 'real world' of reviewing and publishing data. This article outlines a study of the publicly available peer review policies of 39 scientific publications that publish data papers to discern which criteria are most and least frequently referenced. It also compares current practice with proposed criteria published in 2012.
Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determi…
This article shows why the diffusion and peer-reviewing of research results would be more efficient, precise and relevant if all or at least some parts of the descriptions and peer-reviews of research results took the form of a fine-grained semantic network, within articles or knowledge bases, as part of the Semantic Web. This article also shows some ways this can be done and hence how research journal/proceeding publishers could allow this. So far, the World Wide Web Consortium (W3C) has not proposed simple notations and cooperation protocols - similar to those illustrated or referred to in t…
Context: Pre-publication peer review of scientific articles is considered a key element of the research process in software engineering, yet it is often perceived as not to work fully well. Objective: We aim at understanding the perceptions of and attitudes towards peer review of authors and reviewers at one of software engineering's most prestigious venues, the International Conference on Software Engineering (ICSE). Method: We invited 932 ICSE 2014/15/16 authors and reviewers to participate in a survey with 10 closed and 9 open questions. Results: We present a multitude of results, such as: …
The purpose of this study is to analyze the research article publishing with special reference to preparing to publish and peer reviewing. Peer reviewing is the process required for standardizing any publications. Manuscript writing is an art. Though it appears to be simple there is a lot of effort required. Peer reviewing is the process that eliminates articles that do not meet the standard of the journals and the scope of the journals. The study investigated authors views on manuscript submissions to the publishing process. There are 375 samples selected for this study who have experienced p…
Anonymous peer review is used by the great majority of computer science conferences. OpenReview is such a platform that aims to promote openness in peer review process. The paper, (meta) reviews, rebuttals, and final decisions are all released to public. We collect 5,527 submissions and their 16,853 reviews from the OpenReview platform. We also collect these submissions' citation data from Google Scholar and their non-peer-reviewed versions from arXiv.org. By acquiring deep insights into these data, we have several interesting findings that could help understand the effectiveness of the public…
Google Research（谷歌研究院）；University of Cambridge（剑桥大学）
In this paper we revisit the 2014 NeurIPS experiment that examined inconsistency in conference peer review. We determine that 50\% of the variation in reviewer quality scores was subjective in origin. Further, with seven years passing since the experiment we find that for \emph{accepted} papers, there is no correlation between quality scores and impact of the paper as measured as a function of citation count. We trace the fate of rejected papers, recovering where these papers were eventually published. For these papers we find a correlation between quality scores and impact. We conclude that t…
Peer-review system has long been relied upon for bringing quality research to the notice of the scientific community and also preventing flawed research from entering into the literature. The need for the peer-review system has often been debated as in numerous cases it has failed in its task and in most of these cases editors and the reviewers were thought to be responsible for not being able to correctly judge the quality of the work. This raises a question "Can the peer-review system be improved?" Since editors and reviewers are the most important pillars of a reviewing system, we in this w…
Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data
Industrial Economics and Management, KTH Royal Institute of Technology（工业经济与管理，皇家理工学院）；School of Information, Journalism and Communication, University of Sheffield（信息、新闻与传播学院，谢菲尔德大学）
Peer review is the foundation of scientific publication, and the task of reviewing has long been seen as a cornerstone of professional service. However, the massive growth in the field of machine learning has put this community benefit under stress, threatening both the sustainability of an effective review process and the overall progress of the field. In this position paper, we argue that a tragedy of the commons outcome may be avoided by emphasizing the professional aspects of this service. In particular, we propose a rubric to hold reviewers to an objective standard for review quality. In …
In post-publication peer review, scientific contributions are first published in open-access forums, such as arXiv or other digital libraries, and are subsequently reviewed and possibly ranked and/or evaluated. Compared to the classical process of scientific publishing, in which review precedes publication, post-publication peer review leads to faster dissemination of ideas, and publicly-available reviews. The chief concern in post-publication reviewing consists in eliciting high-quality, insightful reviews from participants. We describe the mathematical foundations and structure of TrueReview…
Peer-review is a necessary and essential quality control step for scientific publications but lacks proper incentives. Indeed, the process, which is very costly in terms of time and intellectual investment, not only is not remunerated by the journals but is also not openly recognized by the academic community as a relevant scientific output for a researcher. Therefore, scientific dissemination is affected in timeliness, quality, and fairness. Here, to solve this issue, we propose a blockchain-based incentive system that rewards scientists for peer-reviewing other scientists' work and that buil…
Peer review is widely regarded as essential for advancing scientific research. However, reviewers may be biased by authors' prestige or other characteristics. Double-blind peer review, in which the authors' identities are masked from the reviewers, has been proposed as a way to reduce reviewer bias. Although intuitive, evidence for the effectiveness of double-blind peer review in reducing bias is limited and mixed. Here, we examine the effects of double-blind peer review on prestige bias by analyzing the peer review files of 5027 papers submitted to the International Conference on Learning Rep…
Publishing scientific journals governed by editors relying on anonymous peer reviewing is slow (even one round of reviewing involves several communications between authors, editor and reviewers), partial (arguments of authors can hardly overrule those of reviewers) and not using all available scientific material (even the most thorough and insightful reviews remain for the eyes of authors and editors only). Here I propose a new concept for scientific journals that ensures rapid, impartial and comprehensive (RIC) publishing. RIC concept is based on implementation of two novel publishing princip…
Automated scholarly paper review: Concepts, technologies, and challenges
Xiamen University（厦门大学）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan, Ministry of Culture and Tourism（文化和旅游部闽台非物质文化遗产数字化保护与智能处理重点实验室）
Automatic Reviewers Assignment to a Research Paper Based on Allied References and Publications Weight
Department of Computer Science and Engineering（计算机科学与工程系）；Green University of Bangladesh（孟加拉国绿色大学）；University of Dhaka（达卡大学）；Dhaka International University（达卡国际大学）
