基于神经网络的系统性文献综述引文自动筛选的可重复性研究
Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study
摘要：
在系统性文献综述的过程中，引文筛选被认为是最耗时的步骤之一。已经提出了使用各种机器学习技术来使其自动化的多种方法。将深度神经网络应用于这个问题的第一篇研究论文是在过去两年发表的。在这项工作中，我们对前两篇用于引文筛选的深度学习论文进行了可复制性研究，并在23个公开数据集上评估了它们的性能。虽然我们成功地复制了其中一篇论文的结果，但我们无法复制另一篇论文的结果。我们总结了复制过程中所面临的挑战，包括难以获得与原始论文的实验设置相匹配的数据集，以及执行原始源代码的问题。受此经验的启发，我们随后提出了一个更简单的模型，该模型基于平均词嵌入，在23个数据集中的18个数据集上优于其中一个模型，并且平均比第二种复制方法快72倍。最后，我们测量了模型在各种输入特征和随机初始化下的训练时间和不变性，展示了这些方法的鲁棒性差异。
英文摘要：
In the process of Systematic Literature Review, citation screening is estimated to be one of the most time-consuming steps. Multiple approaches to automate it using various machine learning techniques have been proposed. The first research papers that apply deep neural networks to this problem were published in the last two years. In this work, we conduct a replicability study of the first two deep learning papers for citation screening and evaluate their performance on 23 publicly available datasets. While we succeeded in replicating the results of one of the papers, we were unable to replicate the results of the other. We summarise the challenges involved in the replication, including difficulties in obtaining the datasets to match the experimental setup of the original papers and problems with executing the original source code. Motivated by this experience, we subsequently present a simpler model based on averaging word embeddings that outperforms one of the models on 18 out of 23 datasets and is, on average, 72 times faster than the second replicated approach. Finally, we measure the training time and the invariance of the models when exposed to a variety of input features and random initialisations, demonstrating differences in the robustness of these approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
