arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14602cs.SE
收藏

别提它:一种评估使用软件提及进行引用与可发现性研究所面临挑战的方法

Don't mention it: An approach to assess challenges to using software mentions for citation and discoverability research

Stephan Druskat, Neil P. Chue Hong, Sammie Buzzard, Olexandr Konovalov, Patrick Kornek

AI总结:

本文提出一种评估软件提及数据集可用性的方法,并通过应用于两个数据集揭示了数据集结构、提及质量及软件链接等方面的挑战,指出软件引用实践仍需改进。

摘要:

从学术出版物中收集软件提及的数据集,有可能被用于研究已发表研究中所使用的软件,以及软件引用的实践。最近,一些具有不同特征的新软件提及数据集已经发布。我们提出了一种评估此类数据集在研究软件研究中的可用性的方法。我们的方法包括抽样与数据准备、针对质量和提及特征的人工标注,以及标注分析。我们基于定性观察,将该方法应用于两个软件提及数据集进行评估。通过这样做,我们能够发现使用所选数据集开展研究所面临的挑战。主要问题涉及数据集的结构、所提取提及的质量(分别有54%和23%的提及并非指向软件),以及软件的可访问性。一个数据集完全不提供所提及软件的链接,而另一个数据集提供链接的方式可能会妨碍定量研究工作:(1)链接可能来自不同来源,并且对于同一条提及,每个链接指向不同的软件。(2)自动检索到的链接质量普遍较差(在我们的样本中,65.4%的链接指向了错误的软件)。(3)链接仅存在于一小部分提及中(在我们的样本中为20.5%),这可能导致样本偏斜或比例失衡。然而,在使用软件提及数据集时最大的挑战和根本性问题在于,软件引用的实践仍然不够理想:软件不应仅仅被提及,而应遵循软件引用原则进行引用。

英文摘要:

Datasets collecting software mentions from scholarly publications can potentially be used for research into the software that has been used in the published research, as well as into the practice of software citation. Recently, new software mention datasets with different characteristics have been published. We present an approach to assess the usability of such datasets for research on research software. Our approach includes sampling and data preparation, manual annotation for quality and mention characteristics, and annotation analysis. We applied it to two software mention datasets for evaluation based on qualitative observation. Doing this, we were able to find challenges to working with the selected datasets to do research. Main issues refer to the structure of the dataset, the quality of the extracted mentions (54% and 23% of mentions respectively are not to software), and software accessibility. While one dataset does not provide links to mentioned software at all, the other does so in a way that can impede quantitative research endeavors: (1) Links may come from different sources and each point to different software for the same mention. (2) The quality of the automatically retrieved links is generally poor (in our sample, 65.4% link the wrong software). (3) Links exist only for a small subset (in our sample, 20.5%) of mentions, which may lead to skewed or disproportionate samples. However, the greatest challenge and underlying issue in working with software mention datasets is the still suboptimal practice of software citation: Software should not be mentioned, it should be cited following the software citation principles.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑