Automated Short Answer Grading (ASAG) has been an active area of machine-learning research for over a decade. It promises to let educators grade and give feedback on free-form responses in large-enrollment courses in spite of limited availability of human graders. Over the years, carefully trained models have achieved increasingly higher levels of performance. More recently, pre-trained Large Language Models (LLMs) emerged as a commodity, and an intriguing question is how a general-purpose tool without additional training compares to specialized models. We studied the performance of GPT-4 on the standard benchmark 2-way and 3-way datasets SciEntsBank and Beetle, where in addition to the standard task of grading the alignment of the student answer with a reference answer, we also investigated withholding the reference answer. We found that overall, the performance of the pre-trained general-purpose GPT-4 LLM is comparable to hand-engineered models, but worse than pre-trained LLMs that had specialized training.
LLM-based Automated Grading with Human-in-the-Loop
Computer Science and Engineering(计算机科学与工程);Michigan State University(密歇根州立大学);Rossier School of Education(罗塞尔教育学院);University of Southern California(南加州大学)
GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks
University of California, Santa Barbara(加州大学圣塔芭芭拉分校);University of California, San Diego(加州大学圣迭戈分校);University of Southern California(南加州大学);Bytedance, US(字节跳动(美国))
ChatGPT for automated grading of short answer questions in mechanical ventilation
Shri Atal Bihari Vajpayee Medical College and Research Institute(阿塔尔·比哈里·瓦杰帕伊医学院与研究所);Bowring and Lady Curzon Hospitals(鲍林与莱迪·柯曾医院);Rajiv Gandhi University of Health Sciences(拉吉夫·甘地健康科学大学);Intensive Care Service, Westmead Hospital(韦斯特米德医院重症监护服务部)
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
DIPF | Leibniz Institute for Research and Information in Education(德意志教育研究所 | 列支敦士登教育研究所);Faculty of Computer Science, Goethe University Frankfurt(法兰克福歌德大学计算机学院);Chemnitz University of Technology(化学工业大学)
Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
Quantic School of Business and Technology(Quantic商业与技术学院);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Indonesia University of Education(印度尼西亚教育大学);Monash University(莫纳什大学)
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Universidade de Vigo, Department of Computer Science, ESEI-Higher School of Computer Engineering(维戈大学计算机科学系ESEI高等计算机工程学院);IFCAE-Institute for Research in Physics, Computing and Aerospace Science(IFCAE物理、计算与航空航天科学研究所)
Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
Information Processing and Telecommunications Center Universidad Politécnica de Madrid ETSI Telecomunicación(信息处理与电信中心西班牙马德里理工大学电信工程学院);King’s College London(伦敦国王学院);INGENIO (CSIC-Universitat Politècnica de València)(INGENIO(西班牙国家研究委员会-瓦伦西亚理工大学))
REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading
School of Computing and Augmented Intelligence, Arizona State University, USA(计算与增强智能学院,亚利桑那州立大学,美国);Mary Lou Fulton Teachers College, Arizona State University, USA(玛丽·卢·福洛顿教师学院,亚利桑那州立大学,美国);Department of Computer Science, National Yang Ming Chiao Tung University, TW(国立阳明交通大学计算机科学系,台湾)
FRACTURED-SORRY-Bench: Framework for Revealing Attacks in Conversational Turns Undermining Refusal Efficacy and Defenses over SORRY-Bench (Automated Multi-shot Jailbreaks)
SteLLA: A Structured Grading System Using LLMs with RAG
Department of Computer Science Fitchburg State University(弗itchburg州立大学计算机科学系);Department of Computer Science University of Massachusetts Boston(马萨诸塞大学波士顿分校计算机科学系);Chantilly High School(chantilly高中)
Automated Information Extraction from Thyroid Operation Narrative: A Comparative Study of GPT-4 and Fine-tuned KoELECTRA
Seoul National University(首尔大学);Ewha Womans University Medical Center(梨花女子大学医疗院);Seoul National University Hospital and College of Medicine(首尔大学医院暨医学院)