arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.06752cs.DLcs.AI
收藏

使用大型语言模型评估研究质量:ChatGPT在不同设置和输入下的有效性分析

Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs

University of Sheffield(谢菲尔德大学)

Mike Thelwall

AI总结:

本文研究了ChatGPT在不同输入(全文、标题摘要、仅标题)和模型设置下评估学术文章质量的有效性,发现标题和摘要为最优输入,其评分与人类评分相关性达0.67,且线性回归转换可提升准确性。

摘要:

学术期刊文章质量评估是国家研究评估、任命和晋升中耗时但关键的任务。因此,研究大型语言模型(LLMs)能否在这一过程中发挥作用至关重要。本文评估了ChatGPT的哪些输入(无表格、图表和参考文献的全文;标题和摘要;仅标题)能产生更优的质量评分估计,以及ChatGPT模型和系统提示对评分的影响程度。结果显示,最优输入是文章标题和摘要,基于这些输入的ChatGPT平均评分(在51篇论文的数据集上进行30次迭代)与人类评分的相关性为0.67,是目前报道的最高值。ChatGPT 4o略优于3.5-turbo(0.66)和4o-mini(0.66)。结果表明,尽管针对该任务的复杂系统指令比简单指令更有效,但文章全文可能会干扰LLM的研究质量评估。因此,虽然摘要包含的信息不足以全面评估严谨性,但它们可能包含关于原创性和重要性的强烈提示。最后,线性回归可用于将模型评分转换为人类尺度评分,其准确性比猜测高31%。

英文摘要:

Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. This article assesses which ChatGPT inputs (full text without tables, figures and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts. The results show that the optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones. Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑