Open-ended short-answer questions (SAGs) have been widely recognized as a powerful tool for providing deeper insights into learners' responses in the context of learning analytics (LA). However, SAGs often present challenges in practice due to the high grading workload and concerns about inconsistent assessments. With recent advancements in natural language processing (NLP), automatic short-answer grading (ASAG) offers a promising solution to these challenges. Despite this, current ASAG algorithms are often limited in generalizability and tend to be tailored to specific questions. In this paper, we propose a unified multi-agent ASAG framework, GradeOpt, which leverages large language models (LLMs) as graders for SAGs. More importantly, GradeOpt incorporates two additional LLM-based agents - the reflector and the refiner - into the multi-agent system. This enables GradeOpt to automatically optimize the original grading guidelines by performing self-reflection on its errors. Through experiments on a challenging ASAG task, namely the grading of pedagogical content knowledge (PCK) and content knowledge (CK) questions, GradeOpt demonstrates superior performance in grading accuracy and behavior alignment with human graders compared to representative baselines. Finally, comprehensive ablation studies confirm the effectiveness of the individual components designed in GradeOpt.
LLM-based Automated Grading with Human-in-the-Loop
Computer Science and Engineering(计算机科学与工程);Michigan State University(密歇根州立大学);Rossier School of Education(罗塞尔教育学院);University of Southern California(南加州大学)
Georgia Institute of Technology(佐治亚理工学院);Seoul National University(首尔国立大学);Microsoft(微软);Massachusetts Institute of Technology(麻省理工学院);Brown University(布朗大学)
Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation
University of Florida(佛罗里达大学);National Institute of Technology Calicut(卡利卡特国家理工学院);SRM, Chennai(SRM大学(金奈));University of Cincinnati(辛辛那提大学);Dartmouth College(达特茅斯学院);Arizona State University(亚利桑那州立大学)
SmartCourse: A Contextual AI-Powered Course Advising System for Undergraduates
Department of Computer Science, Wenzhou-Kean University, Wenzhou, China(计算机科学系,温州市凯恩大学,中国温州);Department of Computer Science, Kean University, Union, NJ, United States(计算机科学系,凯恩大学,美国新泽西州联合市)
Streamlining evidence based clinical recommendations with large language models
Engineering Research Center of EMR and Intelligent Expert System, Ministry of Education(教育部电磁辐射与智能专家系统工程研究中心);National Clinical Research Center for Dermatologic and Immunologic Diseases (NCRC-DID), Ministry of Science and Technology(皮肤病与免疫性疾病临床研究中心,科学技术部);State Key Laboratory of Complex Severe and Rare Diseases(复杂严重罕见疾病国家重点实验室);Key Laboratory of Rheumatology and Clinical Immunology, Ministry of Education(风湿免疫学重点实验室,教育部)
Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines
Università degli Studi di Milano Bicocca(米兰比可卡大学);Universitat Pompeu Fabra(庞培法布拉大学);University of Regensburg(雷根斯堡大学);Politecnico di Milano(米兰理工大学);Istituto per le Tecnologie Didattiche (ITD-CNR)(国家研究委员会教育技术研究所);Università degli Studi di Milano(米兰大学);Institute of Science and Technology of Cognition (ISTC-CNR)(国家研究委员会认知科学与技术研究所)
CheckSupport: A Local LLM-Powered Tool for Automated Manuscript Submission Checklist Selection and Completion
Department of Radiology, Perelman School of Medicine at University of Pennsylvania(宾夕法尼亚大学佩雷尔曼医学学院放射科);Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Computer and Information Science, School of Engineering and Applied Science, University of Pennsylvania(宾夕法尼亚大学工程与应用科学学院计算机与信息科学系);Department of Radiology, Cooper University Hospital(科珀大学医院放射科);University of California Davis Graduate School of Management(加州大学戴维斯分校管理研究生院)
Natural Language Programming in Medicine: Administering Evidence Based Clinical Workflows with Autonomous Agents Powered by Generative Large Language Models
Icahn School of Medicine at Mount Sinai(西奈山伊坎医学院);Mount Sinai Hospital(西奈山医院);Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院);University of Pittsburgh School of Medicine(匹兹堡大学医学院)
FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysis
National University of Singapore(国立新加坡大学);Estates Pte Ltd(6Estates公司);Asian Institute of Digital Finance(亚洲数字金融研究所);Hefei University of Technology(合肥工业大学);University of Science and Technology of China(中国科学技术大学)
RECOVER: Designing a Large Language Model-based Remote Patient Monitoring System for Postoperative Gastrointestinal Cancer Care
University of California, Irvine(加州大学尔湾分校);Northeastern University(东北大学);Johns Hopkins University(约翰霍普金斯大学);New York University(纽约大学);MedStar Health Research Institute(梅斯塔健康研究院);Creighton University(克里普顿大学);Georgetown University(乔治城大学)
Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents
Concordia University(康科德大学);Mila - Quebec AI Institute(魁北克人工智能研究所);University of Copenhagen(哥本哈根大学);Universite Claude Bernard Lyon(克莱尔蒙特-伯恩大学);McGill University(麦吉尔大学)
How to Ask the AI: A User Perspective Survey for Large Language Model Prompting
School of Computer Science and Technology, Guangdong University of Technology(广东工业大学计算机科学与技术学院);Department of Computer Science, Hong Kong Baptist University(香港浸会大学计算机科学系)
Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development
Università degli Studi di Milano Bicocca(米兰比可卡大学);University of Regensburg(雷根斯堡大学);Universitat Pompeu Fabra(庞培法布拉大学);Politecnico di Milano(米兰理工大学);National Research Council of Italy (CNR)(意大利国家研究委员会)
Can AI Relate: Testing Large Language Model Response for Mental Health Support
New York University(纽约大学);Grossman School of Medicine(格罗斯曼医学院);Massachusetts Institute of Technology(麻省理工学院);University of California, Los Angeles(加州大学洛杉矶分校)
Evaluating User Experience in Conversational Recommender Systems: A Systematic Review Across Classical and LLM-Powered Approaches
School of Computer Science, University of Technology Sydney(技术科技大学计算机科学学院);The Applied College, King Abdulaziz University(国王阿卜杜勒阿齐兹大学应用学院);Australian Institute of Health Innovation, Macquarie University(麦考瑞大学健康创新研究所)
StorySage: Conversational Autobiography Writing Powered by a Multi-Agent Framework
Department of Management Science and Engineering, Stanford University(管理科学与工程系,斯坦福大学);Department of Electrical Engineering, Stanford University(电气工程系,斯坦福大学);Department of Computer Science, Stanford University(计算机科学系,斯坦福大学)
AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
Tsinghua University(清华大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学)
Inter-Agent Trust Models: A Comparative Study of Brief, Claim, Proof, Stake, Reputation and Constraint in Agentic Web Protocol Design-A2A, AP2, ERC-8004, and Beyond