arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2308.02575cs.CLcs.AI
收藏

GPT-4是可靠的评分者吗?评估GPT-4文本评分的一致性

Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings

University of Passau(帕绍大学)

Veronika Hackl, Alexandra Elena Müller, Michael Granitzer, Maximilian Sailer

AI总结:

本文通过宏观经济任务回答的多轮评分实验,检验GPT-4在时间、迭代和风格变化下的评分一致性,发现其ICC达0.94–0.99且能区分内容与风格。

摘要:

本研究考察了OpenAI的GPT-4(一种最先进的人工智能语言模型)在多次迭代、不同时间跨度和风格变化下生成反馈评分的一致性。该模型从内容和风格方面对高等教育(HE)宏观经济学学科领域内的任务回答进行了评分。研究开展了统计分析,以进一步了解评分者间信度、各次迭代间评分的一致性,以及内容评分与风格评分之间的相关性。结果显示出很高的评分者间信度,不同时间跨度下的ICC分数介于0.94至0.99之间,表明GPT-4在提示清晰的情况下能够在重复评分中生成一致的评分。风格评分与内容评分呈现0.87的高相关性。当采用不恰当的风格时,平均内容评分保持不变,而风格评分下降,这表明该大型语言模型(LLM)在评估过程中能够有效区分这两项标准。此外,本文还介绍并解释了研究中使用的提示。要评估AI模型在各种使用场景中的稳健性和可靠性,仍需进一步研究。

英文摘要:

This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4, a state-of-the-art artificial intelligence language model, across multiple iterations, time spans and stylistic variations. The model rated responses to tasks within the Higher Education (HE) subject domain of macroeconomics in terms of their content and style. Statistical analysis was conducted in order to learn more about the interrater reliability, consistency of the ratings across iterations and the correlation between ratings in terms of content and style. The results revealed a high interrater reliability with ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting that GPT-4 is capable of generating consistent ratings across repetitions with a clear prompt. Style and content ratings show a high correlation of 0.87. When applying a non-adequate style the average content ratings remained constant, while style ratings decreased, which indicates that the large language model (LLM) effectively distinguishes between these two criteria during evaluation. The prompt used in this study is furthermore presented and explained. Further research is necessary to assess the robustness and reliability of AI models in various use cases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑