哪些风格特征会欺骗ChatGPT研究评估?
Which stylistic features fool ChatGPT research evaluations?
AI总结:
本文研究了影响ChatGPT评分的语言因素,发现语言复杂性和长度与ChatGPT评分相关性更强,可能带来偏见。
摘要:
大型语言模型(LLMs)有可能用于支持研究评估,并能中等程度地估计期刊文章的质量。本文评估了是否存在与研究质量无关的语言因素影响ChatGPT的评分。使用99,277篇提交至英国研究卓越框架(REF)2021评估的期刊文章数据集,我们计算了摘要中的可读性指标,并将其与ChatGPT评分和部门REF评分相关联。结果发现,在许多学科领域,语言复杂性和长度与ChatGPT研究质量评分的相关性强于REF专家评分。尽管未测试因果关系,这些结果表明ChatGPT可能比人类专家更倾向于奖励语言复杂性,可能在许多领域偏向于较长且较难阅读的摘要。LLMs对复杂语言的偏好是用于研究质量评估的LLMs的不希望特征,除非能找到解决方案。
英文摘要:
Large Language Models (LLMs) have the potential to be used to support research evaluation and have a moderate capability to estimate the research quality of a journal article from its title and abstract. This paper assesses whether there are language-related factors unrelated to the quality of the research that influence ChatGPT's scores. Using a dataset of 99,277 journal articles submitted to the UK-wide Research Excellence Framework (REF) 2021 assessments, we calculated several readability indicators from abstracts and correlated them with ChatGPT scores and departmental REF scores. From the results, linguistic complexity and length were more strongly associated with ChatGPT research quality scores than with REF expert scores in many subject areas. Although cause-and-effect was not tested, these results suggest that ChatGPT may be more likely than human experts to reward linguistic complexity, with a potential bias towards longer and less readable abstracts in many fields. The apparent preference of LLMs for complex language is an undesirable feature for practical applications of LLMs for research quality evaluation, unless solutions can be found.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。