arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.18924cs.CL
收藏

示例选择在基于GPT模型的小样本提示自动作文评分中的影响

The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models

The University of Tokyo(东京大学)

Lui Yoshida

AI总结:

本研究探讨小样本提示中示例选择对GPT模型自动作文评分性能的影响,发现示例选择对GPT-3.5影响更大,存在多数标签和近因偏差,且2023年6月版GPT-4稳定性和性能最高。

摘要:

本研究探讨了示例选择对使用GPT模型小样本提示进行自动作文评分(AES)性能的影响。我们评估了小样本提示中示例的选择和顺序对多个版本的GPT-3.5和GPT-4模型的影响。我们的实验涉及119个带有不同示例的提示,并计算二次加权卡帕(QWK)以衡量GPT与人类评分者分数之间的一致性。回归分析用于定量评估示例选择引入的偏差。结果表明,示例选择对QWK的影响因模型而异,GPT-3.5比GPT-4更容易受到示例的影响。我们还发现GPT生成的作文分数和QWK中存在多数标签偏差(倾向于偏爱示例中的多数标签)和近因偏差(倾向于偏爱最新示例的标签),这些偏差在GPT-3.5中更为明显。值得注意的是,仔细的示例选择使GPT-3.5模型能够优于一些GPT-4模型。然而,在GPT模型中,2023年6月版本的GPT-4(不是最新模型)表现出最高的稳定性和性能。我们的研究结果为小样本提示中示例选择在AES中的重要性提供了见解,特别是在GPT-3.5模型中,并强调了对每个模型进行单独性能评估的必要性,即使是次要版本。

英文摘要:

This study investigates the impact of example selection on the performance of au-tomated essay scoring (AES) using few-shot prompting with GPT models. We evaluate the effects of the choice and order of examples in few-shot prompting on several versions of GPT-3.5 and GPT-4 models. Our experiments involve 119 prompts with different examples, and we calculate the quadratic weighted kappa (QWK) to measure the agreement between GPT and human rater scores. Regres-sion analysis is used to quantitatively assess biases introduced by example selec-tion. The results show that the impact of example selection on QWK varies across models, with GPT-3.5 being more influenced by examples than GPT-4. We also find evidence of majority label bias, which is a tendency to favor the majority la-bel among the examples, and recency bias, which is a tendency to favor the label of the most recent example, in GPT-generated essay scores and QWK, with these biases being more pronounced in GPT-3.5. Notably, careful example selection enables GPT-3.5 models to outperform some GPT-4 models. However, among the GPT models, the June 2023 version of GPT-4, which is not the latest model, exhibits the highest stability and performance. Our findings provide insights into the importance of example selection in few-shot prompting for AES, especially in GPT-3.5 models, and highlight the need for individual performance evaluations of each model, even for minor versions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑