arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.06222cs.CLcs.IRcs.LG
收藏

基于级联神经网络的科学论文识别

Cascade Neural Ensemble for Identifying Scientifically Sound Articles

Arizona State University(亚利桑那州立大学)

Ashwin Karthik Ambalavanan, Murthy Devarakonda

摘要:

背景:进行系统评价和荟萃分析的一个重要障碍是有效地找到科学合理的相关文章。通常情况下,只有不到1%的文章符合这一要求,这导致了高度不平衡的任务。虽然为这项任务研究了特征工程和早期的神经网络模型,但仍有机会改进结果。研究方法:我们将过滤文章的问题框定为分类任务,并在来自MEDLINE的约5万篇文章的手动注释数据集上训练和测试了SciBERT的几个集成架构,这是科学文章预训练的BERT的变体。由于科学合理的文章是通过一个多步骤的过程,我们提出了一个新的级联合奏类似的选择过程。我们比较了一个单一的集成模型和其他类型的合奏,以及从以前的研究结果的级联合奏的性能。结果如下:级联集成架构实现了0.7505 F测量,与之前提出并在50K文章的选定子集上评估的CNN模型相比,出错率降低了49.1%。在完整的数据集上,级联集成实现了0.7639 F测量,与使用完整数据集的先前研究中报告的最佳性能相比,出错率降低了19.7%。结论:预先训练的上下文编码器神经网络(例如SciBERT)在过滤科学合理的相关文章方面比以前研究的模型和手动创建的搜索过滤器表现更好。级联集成所实现的卓越性能是一个重要的结果,其推广超出了该任务和数据集,并且类似于IR和数据库中的查询优化。

英文摘要:

Background: A significant barrier to conducting systematic reviews and meta-analysis is efficiently finding scientifically sound relevant articles. Typically, less than 1% of articles match this requirement which leads to a highly imbalanced task. Although feature-engineered and early neural networks models were studied for this task, there is an opportunity to improve the results. Methods: We framed the problem of filtering articles as a classification task, and trained and tested several ensemble architectures of SciBERT, a variant of BERT pre-trained on scientific articles, on a manually annotated dataset of about 50K articles from MEDLINE. Since scientifically sound articles are identified through a multi-step process we proposed a novel cascade ensemble analogous to the selection process. We compared the performance of the cascade ensemble with a single integrated model and other types of ensembles as well as with results from previous studies. Results: The cascade ensemble architecture achieved 0.7505 F measure, an impressive 49.1% error rate reduction, compared to a CNN model that was previously proposed and evaluated on a selected subset of the 50K articles. On the full dataset, the cascade ensemble achieved 0.7639 F measure, resulting in an error rate reduction of 19.7% compared to the best performance reported in a previous study that used the full dataset. Conclusion: Pre-trained contextual encoder neural networks (e.g. SciBERT) perform better than the models studied previously and manually created search filters in filtering for scientifically sound relevant articles. The superior performance achieved by the cascade ensemble is a significant result that generalizes beyond this task and the dataset, and is analogous to query optimization in IR and databases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑