SteLLA：一种基于RAG的LLM结构化评分系统
SteLLA: A Structured Grading System Using LLMs with RAG
AI总结：
本研究提出SteLLA系统，结合RAG与结构化问答评估，实现自动简答题评分，在真实生物学考试数据上与人工评分高度一致，并提供细粒度反馈。
摘要：
大型语言模型（LLMs）在许多应用中展现出强大的通用能力。然而，如何使它们成为某些特定任务（如自动简答题评分（ASAG））的可靠工具仍然是一个挑战。我们提出了SteLLA（基于RAG的LLM结构化评分系统），其中a）使用检索增强生成（RAG）方法，通过从基于教师提供的参考答案和评分标准的高度相关且可靠的外部知识中提取结构化信息，专门增强LLM在ASAG任务上的能力；b）LLM对学生答案进行结构化的、基于问答的评估，以提供分析性评分和反馈。我们从一门大学水平的生物学课程中收集了一个包含学生考试答案的真实世界数据集。实验表明，我们提出的系统能够与人工评分者达成实质性一致，同时提供问题中考查的所有知识点的细分评分和反馈。对GPT4生成的反馈进行的定性和错误分析表明，GPT4擅长捕捉事实，但在评分任务中可能倾向于从给定文本中推断出过多的含义，这为在ASAG系统中使用LLM提供了见解。
英文摘要：
Large Language Models (LLMs) have shown strong general capabilities in many applications. However, how to make them reliable tools for some specific tasks such as automated short answer grading (ASAG) remains a challenge. We present SteLLA (Structured Grading System Using LLMs with RAG) in which a) Retrieval Augmented Generation (RAG) approach is used to empower LLMs specifically on the ASAG task by extracting structured information from the highly relevant and reliable external knowledge based on the instructor-provided reference answer and rubric, b) an LLM performs a structured and question-answering-based evaluation of student answers to provide analytical grades and feedback. A real-world dataset that contains students' answers in an exam was collected from a college-level Biology course. Experiments show that our proposed system can achieve substantial agreement with the human grader while providing break-down grades and feedback on all the knowledge points examined in the problem. A qualitative and error analysis of the feedback generated by GPT4 shows that GPT4 is good at capturing facts while may be prone to inferring too much implication from the given text in the grading task which provides insights into the usage of LLMs in the ASAG system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
