arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.05415cs.DLcs.AI
收藏

用机器学习预测文章质量评分:英国研究卓越框架

Predicting article quality scores with machine learning: The UK Research Excellence Framework

University of Wolverhampton(伍尔弗汉普顿大学);Open University(开放大学)

Mike Thelwall, Kayvan Kousha, Mahshid Abdoli, Emma Stuart, Meiko Makita, Paul Wilson, Jonathan Levitt, Petr Knoth, Matteo Cancellieri

AI总结:

本研究评估AI利用文献计量和元数据输入预测文章质量的可行性,基于英国REF2021的84,966篇文章数据,发现随机森林和极端梯度提升算法在医学、物理科学和经济学领域表现最佳,准确率最高超基线42%,但其他领域效果较差。

摘要:

国家研究评估计划和激励方案以前在过于简化的定量指标和耗时的同行评审之间进行选择,有时会由文献计量学提供支持。在此,我们评估人工智能(AI)是否能够提供第三种替代方案,使用更多文献计量和元数据输入来估计文章质量。我们使用提交给英国研究卓越框架2021的84,966篇文章的临时三级REF2021同行评审分数进行了调查,这些文章匹配了2014-18年的Scopus记录并具有实质性摘要。我们发现,医学和物理科学评估单元(UoAs)以及经济学的准确率最高,在最佳情况下比基线高出42%(总体为72%)。这基于1000个文献计量输入和每个UoA中用于训练的一半文章。社会科学、数学、工程、艺术和人文学科UoAs的预测准确率高于基线的幅度要低得多或接近零。在测试的32种算法中,随机森林分类器(标准或序数)和极端梯度提升分类器算法表现最佳。如果将UoAs合并或替换为Scopus的广泛类别,准确率会降低。我们通过主动学习策略和选择算法估计的预测概率较高的文章来提高准确率,但这大大减少了预测的分数数量。

英文摘要:

National research evaluation initiatives and incentive schemes have previously chosen between simplistic quantitative indicators and time-consuming peer review, sometimes supported by bibliometrics. Here we assess whether artificial intelligence (AI) could provide a third alternative, estimating article quality using more multiple bibliometric and metadata inputs. We investigated this using provisional three-level REF2021 peer review scores for 84,966 articles submitted to the UK Research Excellence Framework 2021, matching a Scopus record 2014-18 and with a substantial abstract. We found that accuracy is highest in the medical and physical sciences Units of Assessment (UoAs) and economics, reaching 42% above the baseline (72% overall) in the best case. This is based on 1000 bibliometric inputs and half of the articles used for training in each UoA. Prediction accuracies above the baseline for the social science, mathematics, engineering, arts, and humanities UoAs were much lower or close to zero. The Random Forest Classifier (standard or ordinal) and Extreme Gradient Boosting Classifier algorithms performed best from the 32 tested. Accuracy was lower if UoAs were merged or replaced by Scopus broad categories. We increased accuracy with an active learning strategy and by selecting articles with higher prediction probabilities, as estimated by the algorithms, but this substantially reduced the number of scores predicted.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑