在哪些领域中ChatGPT的评分比引用更符合研究质量？
In which fields do ChatGPT scores align better than citations with research quality?
AI总结：
ChatGPT-4o和ChatGPT-5 mini在多个领域中表现出比引用更高的研究质量相关性，为新研究质量评估提供了有力证据。
摘要：
尽管基于引用的指标被广泛用于研究评估，但它们不适用于最近发表的研究，只反映研究质量的三个常见维度之一，并且在某些社会科学、艺术和人文学科中价值有限。大型语言模型（LLMs）已被证明可以解决一些这些弱点，ChatGPT-4o mini在不完整数据上显示出最 promising 的结果。本文报告了迄今为止规模最大的ChatGPT-4o mini评估，还评估了其更大的兄弟ChatGPT-4o和ChatGPT-5 mini。基于LLM评分与5次重复平均的部门平均质量评分比较，107212篇英国 refereed 期刊文章在34个基于领域的单位评估（UoAs）中，ChatGPT-4o在大多数情况下略优于ChatGPT-4o mini，尽管结合两者的结果比单独使用更好。ChatGPT-4o评分在34个UoAs中的33个中与研究质量呈正相关，其中31个结果在统计上显著。最大的例外是物理学，对于该领域引用更具用处。ChatGPT-4o评分在21个UoAs中比长期引用率有更高的相关性，在26个UoAs中比短期引用率有更高的相关性。ChatGPT-5 mini在整体上甚至有更强的相关性。总之，结果提供了首次大规模证据，表明ChatGPT-4o和ChatGPT-5 mini在作为新研究质量指标来源方面具有竞争力。
英文摘要：
Although citation-based indicators are widely used for research evaluation, they are not useful for recently published research, reflect only one of the three common dimensions of research quality, and have little value in some social sciences, arts and humanities. Large Language Models (LLMs) have been shown to address some of these weaknesses, with ChatGPT-4o mini showing the most promising results, although on incomplete data. This article reports by far the largest scale evaluation of ChatGPT-4o mini yet and also evaluates its larger sibling ChatGPT-4o and ChatGPT-5 mini. Based on comparisons between LLM scores, averaged over 5 repetitions, and departmental average quality scores for 107,212 UK-based refereed journal articles, ChatGPT-4o is marginally better than ChatGPT-4o mini in most of the 34 field-based Units of Assessment (UoAs) tested, although combining both gives better results than either one. ChatGPT-4o scores have a positive correlation with research quality in 33 of the 34 UoAs, with the results being statistically significant in 31. The most substantial exception is Physics, for which citations are more useful. ChatGPT-4o scores had a higher correlation with research quality than long term citation rates in 21 out of 34 UoAs and a higher correlation than short term citation rates in 26 out of 34 UoAs. ChatGPT-5 mini has even stronger correlations overall. In summary, the results give the first large scale evidence that ChatGPT-4o and ChatGPT-5 mini are competitive with citations as new research quality indicator sources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
