使用ChatGPT进行科研评估:它是否存在年龄、国家、长度或领域的偏见?
Research evaluation with ChatGPT: Is it age, country, length, or field biased?
AI总结:
本文研究ChatGPT评估科研论文质量时是否存在偏见,发现评分受发表年份、领域、国家和摘要长度影响,建议对评分按领域和年份标准化并排查短摘要异常。
摘要:
现有部分研究表明,ChatGPT能够根据期刊文章的标题和摘要来评估其质量。这使得利用ChatGPT质量评分(或许可以与基于引用的公式结合使用)来支持科研评估中的同行评审成为可能。然而,尽管ChatGPT会撰写报告来支撑其评分,其内部处理过程实际上是高度不透明的,且其偏见尚不明确。本文探讨了发表年份和领域是否为产生偏见的因素。研究将包含117,650篇文章的数据集提交给ChatGPT 4o-mini,该数据集具有单一学科性质且在期刊间保持平衡,涵盖2003、2008、2013、2018和2023年发表的26个领域。结果显示,平均评分随时间推移而上升,且这并非由作者国籍或标题与摘要长度的变化所致。不同领域和第一作者所属国家之间的结果也存在显著差异。此外,摘要较长的文章往往得分更高,但这可能是因为此类文章本身质量更好,而非ChatGPT分析了更多文本。因此,为了从ChatGPT获得最准确的科研质量评估结果,必须对ChatGPT的评分按领域和年份进行标准化处理,并检查由短摘要文章集引起的异常情况。
英文摘要:
Some research now suggests that ChatGPT can estimate the quality of journal articles from their titles and abstracts. This has created the possibility to use ChatGPT quality scores, perhaps alongside citation-based formulae, to support peer review for research evaluation. Nevertheless, ChatGPT's internal processes are effectively opaque, despite it writing a report to support its scores, and its biases are unknown. This article investigates whether publication date and field are biasing factors. Based on submitting a monodisciplinary journal-balanced set of 117,650 articles from 26 fields published in the years 2003, 2008, 2013, 2018 and 2023 to ChatGPT 4o-mini, the results show that average scores increased over time, and this was not due to author nationality or title and abstract length changes. The results also varied substantially between fields, and first author countries. In addition, articles with longer abstracts tended to receive higher scores, but plausibly due to such articles tending to be better rather than due to ChatGPT analysing more text. Thus, for the most accurate research quality evaluation results from ChatGPT, it is important to normalise ChatGPT scores for field and year and check for anomalies caused by sets of articles with short abstracts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。