arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.01209cs.SEcs.CY
收藏

理解软件工程论文的同行评审

Understanding Peer Review of Software Engineering Papers

Neil A. Ernst, Jeffrey C. Carver, Daniel Mendez, Marco Torchiano

摘要:

同行评审是一项旨在保持科学出版物质量和完整性的关键活动。然而,在实践中,它远非完美。我们的目标是了解评审员,包括那些因评审而获奖的评审员,如何对软件工程论文进行评审,以确定什么是好的评审方法,什么是好的论文。我们首先与软件工程领域备受尊敬的评审员进行了一系列面对面的访谈。然后,我们利用这些采访的结果开发了一份在线调查中使用的问卷,并发送给来自涵盖多个软件工程学科的备受尊敬的场所的评审员,其中一些人因其评审工作而获奖。我们分析了访谈和175名完成在线调查的评论家(包括获奖和未获奖的评论家)的回复。我们报告了几个描述性结果,包括:45%的获奖者每年审查20多篇会议论文,而28%的非获奖者每年审查20多篇会议论文。88%的评审员花在期刊评审上的时间超过两个小时。我们还报告了定性结果。要写一篇好的评论,重要的标准是它应该是真实的和有帮助的,高于其他的,如详细或善良。获得正面评价的论文最重要的特征是清晰和支持的验证,有趣的问题和新颖性。相反,负面评论往往来自方法和主张之间不匹配的论文,以及那些过于夸大的主张。对作者的主要建议是在论文中非常清楚地说明工作的贡献。此外,审查人员认为数据的提供及其一致性很重要。

英文摘要:

Peer review is a key activity intended to preserve the quality and integrity of scientific publications. However, in practice it is far from perfect. We aim at understanding how reviewers, including those who have won awards for reviewing, perform their reviews of software engineering papers to identify both what makes a good reviewing approach and what makes a good paper. We first conducted a series of in-person interviews with well-respected reviewers in the software engineering field. Then, we used the results of those interviews to develop a questionnaire used in an online survey and sent out to reviewers from well-respected venues covering a number of software engineering disciplines, some of whom had won awards for their reviewing efforts. We analyzed the responses from the interviews and from 175 reviewers who completed the online survey (including both reviewers who had won awards and those who had not). We report on several descriptive results, including: 45% of award-winners are reviewing 20+ conference papers a year, while 28% of non-award winners conduct that many. 88% of reviewers are taking more than two hours on journal reviews. We also report on qualitative results. To write a good review, the important criteria were it should be factual and helpful, ranked above others such as being detailed or kind. The most important features of papers that result in positive reviews are clear and supported validation, an interesting problem, and novelty. Conversely, negative reviews tend to result from papers that have a mismatch between the method and the claims and from those with overly grandiose claims. The main recommendation for authors is to make the contribution of the work very clear in their paper. In addition, reviewers viewed data availability and its consistency as being important.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑