GPT-4与人工评分者在评估合成对话中给予学生的表扬方面的比较分析
Comparative Analysis of GPT-4 and Human Graders in Evaluating Praise Given to Students in Synthetic Dialogues
AI总结：
本文比较了GPT-4与人工评分者在识别辅导对话中有效表扬成分的表现，发现两种提示方法结果相近，但GPT-4在识别真诚表扬方面存在不足，未来将改进提示工程和评分标准。
摘要：
研究表明，向人类辅导者提供具体且及时的反馈能够提升其表现。然而，由于人工评估者评估辅导者表现的过程耗时较长，这一做法面临挑战。诸如AI聊天机器人ChatGPT这样的大型语言模型，有潜力在实际场景中为辅导者提供建设性反馈。尽管如此，AI生成反馈的准确性仍不确定，且很少有研究考察ChatGPT等模型提供有效反馈的能力。在这项进行中的工作中，我们评估了GPT-4在辅导者-学生场景下生成的30段对话。我们使用两种不同的提示方法，即零样本思维链和少样本思维链，基于五项标准来识别有效表扬的具体组成部分。随后，将这些方法的结果与人工评分者的结果进行准确性比较。我们的目标是评估GPT-4能够在多大程度上准确识别每项表扬标准。我们发现，零样本思维链和少样本思维链方法产生了可比较的结果。GPT-4在识别辅导者提供具体且即时表扬的实例方面表现中等。然而，GPT-4在识别辅导者表达真诚表扬的能力方面表现不佳，尤其是在零样本提示场景中，因为该场景未提供真诚辅导者表扬语句的示例。未来的工作将侧重于改进提示工程、开发更通用的辅导评分标准，以及使用真实辅导对话来评估我们的方法。
英文摘要：
Research suggests that providing specific and timely feedback to human tutors enhances their performance. However, it presents challenges due to the time-consuming nature of assessing tutor performance by human evaluators. Large language models, such as the AI-chatbot ChatGPT, hold potential for offering constructive feedback to tutors in practical settings. Nevertheless, the accuracy of AI-generated feedback remains uncertain, with scant research investigating the ability of models like ChatGPT to deliver effective feedback. In this work-in-progress, we evaluate 30 dialogues generated by GPT-4 in a tutor-student setting. We use two different prompting approaches, the zero-shot chain of thought and the few-shot chain of thought, to identify specific components of effective praise based on five criteria. These approaches are then compared to the results of human graders for accuracy. Our goal is to assess the extent to which GPT-4 can accurately identify each praise criterion. We found that both zero-shot and few-shot chain of thought approaches yield comparable results. GPT-4 performs moderately well in identifying instances when the tutor offers specific and immediate praise. However, GPT-4 underperforms in identifying the tutor's ability to deliver sincere praise, particularly in the zero-shot prompting scenario where examples of sincere tutor praise statements were not provided. Future work will focus on enhancing prompt engineering, developing a more general tutoring rubric, and evaluating our method using real-life tutoring dialogues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
