arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1610.00192cs.IRcs.LG
收藏

基于SVM的系统评价摘要筛选方法的大规模研究

A large scale study of SVM based methods for abstract screening in systematic reviews

Indiana University Purdue University Indianapolis(印第安纳大学与普渡大学印第安纳波利斯联合分校);Qatar Computing Research Institute, Hamad-Bin Khalifa University(哈马德·本·哈利法大学卡塔尔计算研究所);University of Lille(里尔大学)

Tanay Kumar Saha, Mourad Ouzzani, Hossam M. Hammady, Ahmed K. Elmagarmid, Wajdi Dhifli, Mohammad Al Hasan

摘要:

系统评价的一项主要任务是摘要筛选,即,基于标题和摘要排除从数据库检索返回的通常成百上千的不相关引文。因此,一个可以自动化摘要筛选过程的系统性综述平台非常重要。已经为这项任务提出了几种方法。然而,由于以下挑战,很难清楚地了解这些方法在系统评价平台中的适用性:(1)使用非重叠度量来评估所提出的方法,(2)使用非常难以收集的特征,(3)使用一小组评论来评估,以及(4)没有可靠的统计检验或方法的等效分组。在本文中,我们使用的特征表示,可以提取每个引用。我们评估基于SVM的方法(常用)的一个大的审查($61$)和指标($11$),以提供基于坚实的统计检验的方法的等效分组。我们的分析还包括使用$500$ x $2$交叉验证的度量的强可变性。虽然有些方法适用于不同的指标和不同的数据集,但没有一种方法可以在其中占据主导地位。此外,我们观察到,在某些情况下,通过基于确定性的抽样,仅筛选其中的15-20%后,可以找到相关(包括)引文。少数纳入的引文存在异常特征,只能在大量筛选步骤后才能找到。最后,我们提出了一个集成算法产生一个$5$星级的引文的相关性的基础上。这种算法结合了我们评估的最佳方法,并通过其$5$星级评级输出更易于消费的预测。

英文摘要:

A major task in systematic reviews is abstract screening, i.e., excluding, often hundreds or thousand of, irrelevant citations returned from a database search based on titles and abstracts. Thus, a systematic review platform that can automate the abstract screening process is of huge importance. Several methods have been proposed for this task. However, it is very hard to clearly understand the applicability of these methods in a systematic review platform because of the following challenges: (1) the use of non-overlapping metrics for the evaluation of the proposed methods, (2) usage of features that are very hard to collect, (3) using a small set of reviews for the evaluation, and (4) no solid statistical testing or equivalence grouping of the methods. In this paper, we use feature representation that can be extracted per citation. We evaluate SVM-based methods (commonly used) on a large set of reviews ($61$) and metrics ($11$) to provide equivalence grouping of methods based on a solid statistical test. Our analysis also includes a strong variability of the metrics using $500$x$2$ cross validation. While some methods shine for different metrics and for different datasets, there is no single method that dominates the pack. Furthermore, we observe that in some cases relevant (included) citations can be found after screening only 15-20% of them via a certainty based sampling. A few included citations present outlying characteristics and can only be found after a very large number of screening steps. Finally, we present an ensemble algorithm for producing a $5$-star rating of citations based on their relevance. Such algorithm combines the best methods from our evaluation and through its $5$-star rating outputs a more easy-to-consume prediction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑