ChatGPT 能否评估研究质量?
Can ChatGPT evaluate research quality?
AI总结:
本文通过基于英国 REF 2021 标准构建研究评估 ChatGPT 并对 51 篇文章进行测试,发现 ChatGPT-4 单次评分与作者自评相关性较弱,但多次迭代平均分相关性显著提升,表明其尚不足以可靠用于正式研究质量评估。
摘要:
目的:评估 ChatGPT 4.0 是否足够准确,能够对期刊文章进行研究评估,从而使这项耗时的工作实现自动化。设计/方法/路径:以已发表的英国 2021 年研究卓越框架(REF)评分指南为案例,测试 ChatGPT-4 在多大程度上能够评估期刊文章的质量,并据此构建一个研究评估 ChatGPT。将该评估应用于我自己的 51 篇文章,并与我自己的质量判断进行比较。发现:ChatGPT-4 能够生成与 REF 标准相符的、看似合理的文档摘要和质量评估理由。其总体得分与我对相同文档的自我评估得分之间仅存在弱相关(15 次迭代平均 r=0.281,其中 8 次与 0 存在统计显著差异)。相比之下,15 次迭代的平均得分产生了统计显著的正相关,相关系数为 0.509。因此,对多轮 ChatGPT-4 的得分取平均值似乎比单次得分更有效。这种正相关可能源于 ChatGPT 能够从每篇论文内部提取作者关于重要性、严谨性和原创性的主张。如果剔除我最弱的文章,则与平均得分的相关性(r=0.200)降至统计显著性以下,这表明 ChatGPT 难以进行细粒度的评估。研究局限:数据来自某一学科中一位学者对其方便样本文章的自我评估。实践意义:总体而言,ChatGPT 似乎尚不够准确,不足以被信任用于任何正式或非正式的研究质量评估任务。因此,包括期刊编辑在内的研究评估者应当采取措施控制其使用。原创性/价值:这是首次公开发表的针对 ChatGPT 的发表后专家评审准确性测试。
英文摘要:
Purpose: Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Design/methodology/approach: Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements. Findings: ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author's significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations. Research limitations: The data is self-evaluations of a convenience sample of articles from one academic in one field. Practical implications: Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use. Originality/value: This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。