Identifying the Development and Application of Artificial Intelligence in Scientific Text
James Dunham, Jennifer Melot, Dewey Murdick
备注:
This revision expands our analysis in Section 5. We predict and evaluate labels for publications in Microsoft Academic Graph and Digital Science Dimensions, in addition to Clarivate Web of Science
我们描述了一种策略,用于识别与人工智能的应用和发展相关的研究出版物的宇宙。这种方法利用了arXiv的科学预印本语料库,作者可以从编辑定义的集合中为论文选择主题标签。我们通过从论文元数据中学习这些主题,然后在较大的语料库中推断论文的arXiv主题标签来组成AI相关性的功能定义:Clarivate Web of Science,Digital Science Dimensions和Microsoft Academic Graph。这为自然语言处理(cs.CL)、计算机视觉(cs.CV)和机器人技术(cs.RO)产生了介于.75和.86之间的预测分类$F_1$分数。对于学习这些和其他四个AI相关主题(cs.AI,cs.LG,stat.ML和cs.MA)的单个模型,我们看到精确率为0.83,召回率为0.85。我们评估我们的分类器对其他来源的主题信息和预测的替代方法的域外性能。我们发现,一个监督的解决方案可以推广到识别属于高级研究领域的arXiv上表示的出版物。这提供了一种识别AI相关出版物的方法,这些出版物以研究输出的速度更新,而无需依赖主题专家进行查询开发或标记。
英文摘要:
We describe a strategy for identifying the universe of research publications relevant to the application and development of artificial intelligence. The approach leverages the arXiv corpus of scientific preprints, in which authors choose subject tags for their papers from a set defined by editors. We compose a functional definition of AI relevance by learning these subjects from paper metadata, and then inferring the arXiv-subject labels of papers in larger corpora: Clarivate Web of Science, Digital Science Dimensions, and Microsoft Academic Graph. This yields predictive classification $F_1$ scores between .75 and .86 for Natural Language Processing (cs.CL), Computer Vision (cs.CV), and Robotics (cs.RO). For a single model that learns these and four other AI-relevant subjects (cs.AI, cs.LG, stat.ML, and cs.MA), we see precision of .83 and recall of .85. We evaluate the out-of-domain performance of our classifiers against other sources of topic information and predictions from alternative methods. We find that a supervised solution can generalize to identify publications that belong to the high-level fields of study represented on arXiv. This offers a method for identifying AI-relevant publications that updates at the pace of research output, without reliance on subject-matter experts for query development or labeling.