使用LLM评估学生的开放式书面回答：将RAG框架用于GPT-3.5、GPT-4、Claude-3和Mistral-Large
Evaluating Students' Open-ended Written Responses with LLMs: Using the RAG Framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large
AI总结：
本文利用RAG框架比较GPT-3.5、GPT-4、Claude-3和Mistral-Large在两种温度下重复评分大学生开放式回答的表现，发现其一致性与评分结果存在明显差异。
摘要：
评估学生的开放式笔试回答对教育工作者来说是一项必不可少但耗时的任务，需要投入大量精力，并保持高度的一致性和精确性。大型语言模型（LLM）的最新发展为兼顾全面评估需求与教育工作者时间的高效利用提供了一个颇有前景的机会。在本研究中，我们考察了ChatGPT-3.5、ChatGPT-4、Claude-3和Mistral-Large等LLM在评估大学生针对其已学习参考材料所出问题的开放式回答时的有效性。每个模型都被要求在两种条件下反复评估54份回答：在温度设置为0.0时评估10次（10-shot），以及在温度为0.5时评估10次；预计每个模型共完成1,080次评估，所有模型合计完成4,320次评估。研究使用RAG（Retrieval Augmented Generation，检索增强生成）框架作为使LLM处理回答评估的框架。截至2024年春季，我们的分析显示，所研究LLM提供的评分结果及其一致性存在显著差异。有必要理解LLM在教育场景中评估开放式书面回答的优势与弱点。为确定使用LLM开展教育评估的准确性和成本效益，进一步的比较研究至关重要。
英文摘要：
Evaluating open-ended written examination responses from students is an essential yet time-intensive task for educators, requiring a high degree of effort, consistency, and precision. Recent developments in Large Language Models (LLMs) present a promising opportunity to balance the need for thorough evaluation with efficient use of educators' time. In our study, we explore the effectiveness of LLMs ChatGPT-3.5, ChatGPT-4, Claude-3, and Mistral-Large in assessing university students' open-ended answers to questions made about reference material they have studied. Each model was instructed to evaluate 54 answers repeatedly under two conditions: 10 times (10-shot) with a temperature setting of 0.0 and 10 times with a temperature of 0.5, expecting a total of 1,080 evaluations per model and 4,320 evaluations across all models. The RAG (Retrieval Augmented Generation) framework was used as the framework to make the LLMs to process the evaluation of the answers. As of spring 2024, our analysis revealed notable variations in consistency and the grading outcomes provided by studied LLMs. There is a need to comprehend strengths and weaknesses of LLMs in educational settings for evaluating open-ended written responses. Further comparative research is essential to determine the accuracy and cost-effectiveness of using LLMs for educational assessments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
