arXiv 2505.155962025-05-21cs.HCcs.AI

探索LLM生成的经济学论文反馈：助教如何评估并设想其使用

Exploring LLM-Generated Feedback for Economics Essays: How Teaching Assistants Evaluate and Envision Its Use

University of Michigan（密歇根大学）

Xinyi Lu, Aditya Mahesh, Zejia Shen, Mitchell Dudley, Larissa Sano, Xu Wang

备注：: To be published in AIED'2025: In Proceedings of the 26th International Conference on Artificial Intelligence in Education. The system prompt and example feedback can be found through http://github.com/UM-Lifelong-Learning-Lab/AIED2025-Exploring-LLM-Generated-Feedback-for-Economics-Essay

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究通过助教评估，探索LLM生成反馈在经济学论文评分中的应用，发现详细评分标准可提升反馈质量，且AI建议能加快评分、增强一致性，并强调分解任务步骤以促进使用。

摘要：

本项目探讨了使用AI生成的反馈作为建议，以加快并增强人类教师反馈提供的可能性。我们特别关注理解助教对AI生成反馈质量的看法，以及他们是否以及如何在自身工作流程中利用AI反馈。我们将工作置于一门基础大学经济学课程中，该课程有频繁的短文作业。我们开发了一个基于LLM的反馈引擎，根据助教使用的评分标准生成对学生论文的反馈。为了确保助教能够有意义地批评和参与AI反馈，我们让他们完成常规的评分工作。对于他们已评分的随机选择的论文集，我们使用反馈引擎生成反馈，并将反馈以Word文档中的文本内注释形式显示。随后，我们对5名助教进行了20次1小时的出声思考研究，让他们评估AI反馈，将AI反馈与他们的手写反馈进行对比，并分享他们如果AI反馈作为建议提供时如何设想使用它。研究强调了提供详细评分标准对于AI生成高质量反馈（针对知识密集型论文）的重要性。助教认为，在评分过程中使用AI反馈作为建议可以加快评分、增强一致性并提高整体反馈质量。我们讨论了将反馈生成任务分解为步骤并呈现中间结果的重要性，以便助教使用AI反馈。

英文摘要：

This project examines the prospect of using AI-generated feedback as suggestions to expedite and enhance human instructors' feedback provision. In particular, we focus on understanding the teaching assistants' perspectives on the quality of AI-generated feedback and how they may or may not utilize AI feedback in their own workflows. We situate our work in a foundational college Economics class, which has frequent short essay assignments. We developed an LLM-powered feedback engine that generates feedback on students' essays based on grading rubrics used by the teaching assistants (TAs). To ensure that TAs can meaningfully critique and engage with the AI feedback, we had them complete their regular grading jobs. For a randomly selected set of essays that they had graded, we used our feedback engine to generate feedback and displayed the feedback as in-text comments in a Word document. We then performed think-aloud studies with 5 TAs over 20 1-hour sessions to have them evaluate the AI feedback, contrast the AI feedback with their handwritten feedback, and share how they envision using the AI feedback if they were offered as suggestions. The study highlights the importance of providing detailed rubrics for AI to generate high-quality feedback for knowledge-intensive essays. TAs considered that using AI feedback as suggestions during their grading could expedite grading, enhance consistency, and improve overall feedback quality. We discuss the importance of decomposing the feedback generation task into steps and presenting intermediate results, in order for TAs to use the AI feedback.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。