Literature reviews allow scientists to stand on the shoulders of giants, showing promising directions, summarizing progress, and pointing out existing challenges in research. At the same time conducting a systematic literature review is a laborious and consequently expensive process. In the last decade, there have a few studies on crowdsourcing in literature reviews. This paper explores the feasibility of crowdsourcing for facilitating the literature review process in terms of results, time and effort, as well as to identify which crowdsourcing strategies provide the best results based on the budget available. In particular we focus on the screening phase of the literature review process and we contribute and assess methods for identifying the size of tests, labels required per paper, and classification functions as well as methods to split the crowdsourcing process in phases to improve results. Finally, we present our findings based on experiments run on Crowdflower.
A large scale study of SVM based methods for abstract screening in systematic reviews
Indiana University Purdue University Indianapolis(印第安纳大学与普渡大学印第安纳波利斯联合分校);Qatar Computing Research Institute, Hamad-Bin Khalifa University(哈马德·本·哈利法大学卡塔尔计算研究所);University of Lille(里尔大学)
系统评价的一项主要任务是摘要筛选,即,基于标题和摘要排除从数据库检索返回的通常成百上千的不相关引文。因此,一个可以自动化摘要筛选过程的系统性综述平台非常重要。已经为这项任务提出了几种方法。然而,由于以下挑战,很难清楚地了解这些方法在系统评价平台中的适用性:(1)使用非重叠度量来评估所提出的方法,(2)使用非常难以收集的特征,(3)使用一小组评论来评估,以及(4)没有可靠的统计检验或方法的等效分组。在本文中,我们使用的特征表示,可以提取每个引用。我们评估基于SVM的方法(常用)的一个大的审查($61$)和指标($11$),以提供基于坚实的统计检验的方法的等效分组。我们的分析还包括使用$500$ x $2$交叉验证的度量的强可变性。虽然有些方法适用于不同的指标和不同的数据集,但没有一种方法可以在其中占据主导地位。此外,我们观察到,在某些情况下,通过基于确定性的抽样,仅筛选其中的15-20%后,可以找到相关(包括)引文。少数纳入的引文存在异常特征,只能在大量筛选步骤后才能找到。最后,我们提出了一个集成算法产生一个$5$星级的引文的相关性的基础上。这种算法结合了我们评估的最佳方法,并通过其$5$星级评级输出更易于消费的预测。
ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews
University of Arizona(亚利桑那大学);Florida International University(佛罗里达国际大学);Vanderbilt University(范德堡大学);University of California, Irvine(加州大学欧文分校);New York University(纽约大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所);University of Cincinnati(辛辛那提大学);University of South Florida(南佛罗里达大学);Vanderbilt University Medical Center(范德堡大学医学中心)
High-performance automated abstract screening with large language model ensembles
University of Oxford(牛津大学);Oxford University Hospitals NHS Foundation Trust(牛津大学医院国民保健服务基金会信托);University of Cambridge(剑桥大学);Georgetown University(乔治城大学);MedStar Washington Hospital Center(MedStar华盛顿医院中心);King’s College London(伦敦国王学院);University College London(伦敦大学学院)