Every four years, the PISA test is administered by the OECD to test the knowledge of teenage students worldwide and allow for comparisons of educational systems. However, having to avoid language differences and annotator bias makes the grading of student answers challenging. For these reasons, it would be interesting to consider methods of automatic student answer grading. To train some of these methods, which require machine learning, or to compute parameters or select hyperparameters for those that do not, a large amount of domain-specific data is needed. In this work, we explore a small number of methods for creating a large-scale training dataset using only a relatively small confidential dataset as a reference, leveraging a set of very simple derived text formats to preserve confidentiality. Using the proposed methods, we successfully created three surrogate datasets that are, at the very least, superficially more similar to the reference dataset than a straightforward result of prompt-based generation. Early experiments suggest one of these approaches might also lead to improved training of automatic answer grading models.
Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
DIPF | Leibniz Institute for Research and Information in Education（DIPF | 莱布尼茨教育与信息研究所）；IPN | Leibniz Institute for Science and Mathematics Education（IPN | 莱布尼茨科学与数学教育研究所）；Umeå University（于默奥大学）；Goethe University Frankfurt（法兰克福大学）
Meta Superintelligence Labs（Meta超智能实验室）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Oxford（牛津大学）；University of Cambridge（剑桥大学）
LLM-based Automated Grading with Human-in-the-Loop
Computer Science and Engineering（计算机科学与工程）；Michigan State University（密歇根州立大学）；Rossier School of Education（罗塞尔教育学院）；University of Southern California（南加州大学）
REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading
School of Computing and Augmented Intelligence, Arizona State University, USA（计算与增强智能学院，亚利桑那州立大学，美国）；Mary Lou Fulton Teachers College, Arizona State University, USA（玛丽·卢·福洛顿教师学院，亚利桑那州立大学，美国）；Department of Computer Science, National Yang Ming Chiao Tung University, TW（国立阳明交通大学计算机科学系，台湾）
Rubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring
DIPF | Leibniz Institute for Research and Information in Education（DIPF | 莱布尼茨教育与信息研究所）；Centre for International Student Assessment (ZIB)（国际学生评估中心（ZIB））；Goethe University Frankfurt（法兰克福歌德大学）
提出 RUSPAN 框架，将评分标准视为标签表示并联合编码问题、回答与标准级别，通过评分标准无关掩码实现零样本迁移，在六种语言的 ASAS 基准上超越基线，并在 PT-ASAG 上取得显著提升。
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
DIPF | Leibniz Institute for Research and Information in Education（德意志教育研究所 | 列支敦士登教育研究所）；Faculty of Computer Science, Goethe University Frankfurt（法兰克福歌德大学计算机学院）；Chemnitz University of Technology（化学工业大学）
ChatGPT for automated grading of short answer questions in mechanical ventilation
Shri Atal Bihari Vajpayee Medical College and Research Institute（阿塔尔·比哈里·瓦杰帕伊医学院与研究所）；Bowring and Lady Curzon Hospitals（鲍林与莱迪·柯曾医院）；Rajiv Gandhi University of Health Sciences（拉吉夫·甘地健康科学大学）；Intensive Care Service, Westmead Hospital（韦斯特米德医院重症监护服务部）
Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach
Universidade de Vigo, Department of Computer Science, ESEI-Higher School of Computer Engineering（维戈大学计算机科学系ESEI高等计算机工程学院）；IFCAE-Institute for Research in Physics, Computing and Aerospace Science（IFCAE物理、计算与航空航天科学研究所）
Confidence Estimation in Automatic Short Answer Grading with LLMs
DIPF | Leibniz Institute for Research and Information in Education（莱布尼茨教育研究与信息研究所）；Faculty of Computer Science, Goethe University Frankfurt（弗赖堡大学计算机科学系）；Chemnitz University of Technology（化学工业大学）
Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China（数据科学学院，香港中文大学（深圳））；UC Berkeley（加州大学伯克利分校）；Zhejiang University（浙江大学）；Johns Hopkins University（约翰霍普金斯大学）；Renmin University of China（中国人民大学）；Xiaohongshu Inc.（小红书公司）
Dr.Credit: Rubric-Grounded Process Credit Assignment for Deep Research Agents
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Alibaba Group（阿里巴巴集团）；Peking University（北京大学）
