Grammatical error detection and automated essay scoring are two tasks in the area of automated assessment. Traditionally these tasks have been treated independently with different machine learning models and features used for each task. In this paper, we develop a multi-task neural network model that jointly optimises for both tasks, and in particular we show that neural automated essay scoring can be significantly improved. We show that while the essay score provides little evidence to inform grammatical error detection, the essay score is highly influenced by error detection.
Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
Quantic School of Business and Technology（Quantic商业与技术学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Indonesia University of Education（印度尼西亚教育大学）；Monash University（莫纳什大学）
COMSATS Institute of Information Technology（COMSATS信息技术学院）
Assessments help in evaluating the knowledge gained by a learner at any specific point as well as in continuous improvement of the curriculum design and the whole learning process. However, with the increase in students' enrollment at University level in either conventional or distance education environment, traditional ways of assessing students' work are becoming insufficient in terms of both time and effort. In distance education environment, such assessments become additionally more challenging in terms of hefty remuneration for hiring large number of tutors. The availability of automated …
Automated question generation is an important approach to enable personalisation of English comprehension assessment. Recently, transformer-based pretrained language models have demonstrated the ability to produce appropriate questions from a context paragraph. Typically, these systems are evaluated against a reference set of manually generated questions using n-gram based metrics, or manual qualitative assessment. Here, we focus on a fully automated multiple-choice question generation (MCQG) system where both the question and possible answers must be generated from the context paragraph. Appl…
Learnable Assessment Skills for LLM-based Automated Scoring: Rubric Construction via Iterative Optimization
School of Computing, University of Georgia, Athens, GA, USA（佐治亚大学计算机学院）；AI4STEM Education Center, University of Georgia, Athens, GA, USA（佐治亚大学AI4STEM教育中心）；The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学）
Strategy training is a multidisciplinary rehabilitation approach that teaches skills to reduce disability among those with cognitive impairments following a stroke. Strategy training has been shown in randomized, controlled clinical trials to be a more feasible and efficacious intervention for promoting independence than traditional rehabilitation approaches. A standardized fidelity assessment is used to measure adherence to treatment principles by examining guided and directed verbal cues in video recordings of rehabilitation sessions. Although the fidelity assessment for detecting guided and…
Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation
School of Foreign Languages, Shanghai Jiao Tong University（上海交通大学外国语学院）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；Institute of Corpus Studies and Applications, Shanghai International Studies University（上海外国语大学语料库研究与应用研究所）
Grammatical Error Correction (GEC) systems perform a sequence-to-sequence task, where an input word sequence containing grammatical errors, is corrected for these errors by the GEC system to output a grammatically correct word sequence. With the advent of deep learning methods, automated GEC systems have become increasingly popular. For example, GEC systems are often used on speech transcriptions of English learners as a form of assessment and feedback - these powerful GEC systems can be used to automatically measure an aspect of a candidate's fluency. The count of \textit{edits} from a candid…
Technische Hochschule Nürnberg Georg Simon Ohm（纽伦堡乔治·西蒙·欧姆应用科学大学）；Institut der Kasseler Stottertherapie（卡塞尔口吃治疗研究所）
Stuttering is a complex speech disorder that can be identified by repetitions, prolongations of sounds, syllables or words, and blocks while speaking. Severity assessment is usually done by a speech therapist. While attempts at automated assessment were made, it is rarely used in therapy. Common methods for the assessment of stuttering severity include percent stuttered syllables (% SS), the average of the three longest stuttering symptoms during a speech task, or the recently introduced Speech Efficiency Score (SES). This paper introduces the Speech Control Index (SCI), a new method to evalua…
Sentiment analysis is a sub-discipline in the field of natural language processing and computational linguistics and can be used for automated or semi-automated analyses of text documents. One of the aims of these analyses is to recognize an expressed attitude as positive or negative as it can be contained in comments on social media platforms or political documents and speeches as well as fictional and nonfictional texts. Regarding analyses of comments on social media platforms, this is an extension of the previous tutorial on semi-automated screenings of social media network data. A longitud…
Toward LLM-Supported Automated Assessment of Critical Thinking Subskills
University of Minnesota（明尼苏达大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Adelaide University（阿德莱德大学）；Brighter Research（Brighter研究公司）；Educational Testing Service (ETS)（教育考试服务中心（ETS））
