GPT-4在设计类大学生作业评分与反馈中的应用:一项探索性研究
The application of GPT-4 in grading design university students' assignment and providing feedback: An exploratory study
AI总结:
本研究探讨GPT-4能否有效为设计类大学生作业评分并提供反馈,通过迭代开发Custom GPT,发现其评分者间信度达教育界接受水平,评分者内信度在0.65-0.78间,能提供一致且有用的反馈,可作为人类评分者的可靠补充。
摘要:
本研究旨在调查GPT-4能否有效为设计类大学生作业评分并提供有用的反馈。在设计教育中,作业没有唯一正确答案,通常涉及解决开放式设计问题。设计项目的这种主观性常导致评分问题,因为不同评分者(例如工程背景或建筑背景的教师)给出的分数可能不同。本研究采用迭代研究方法开发Custom GPT,旨在获得更可靠的结果并测试其能否为设计学生提供建设性反馈。研究结果包括:首先,经过几轮迭代,GPT与人类评分者之间的评分者间信度达到了教育工作者普遍接受的水平。这表明,通过向GPT提供准确的提示,并持续迭代构建Custom GPT,可将其有效用于学生设计作业评分,作为人类评分者的可靠补充。其次,GPT在不同时间评分的评分者内信度在0.65至0.78之间。这表明,在充分指令下,Custom GPT能给出一致结果,这是为学生评分的前提。由于一致性和可比性是确保教育评估可靠性的两大原则,本研究探讨了能否开发出遵循这两大原则的Custom GPT。最后,我们测试了Custom GPT能否为学生提供有用反馈,并反思了教育工作者如何开发和迭代Custom GPT以作为补充评分者。
英文摘要:
This study aims to investigate whether GPT-4 can effectively grade assignments for design university students and provide useful feedback. In design education, assignments do not have a single correct answer and often involve solving an open-ended design problem. This subjective nature of design projects often leads to grading problems,as grades can vary between different raters,for instance instructor from engineering background or architecture background. This study employs an iterative research approach in developing a Custom GPT with the aim of achieving more reliable results and testing whether it can provide design students with constructive feedback. The findings include: First,through several rounds of iterations the inter-reliability between GPT and human raters reached a level that is generally accepted by educators. This indicates that by providing accurate prompts to GPT,and continuously iterating to build a Custom GPT, it can be used to effectively grade students' design assignments, serving as a reliable complement to human raters. Second, the intra-reliability of GPT's scoring at different times is between 0.65 and 0.78. This indicates that, with adequate instructions, a Custom GPT gives consistent results which is a precondition for grading students. As consistency and comparability are the two main rules to ensure the reliability of educational assessment, this study has looked at whether a Custom GPT can be developed that adheres to these two rules. We finish the paper by testing whether Custom GPT can provide students with useful feedback and reflecting on how educators can develop and iterate a Custom GPT to serve as a complementary rater.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。