Large language models have demonstrated exceptional capabilities in tasks involving natural language generation, reasoning, and comprehension. This study aims to construct prompts and comments grounded in the diverse scoring criteria delineated within the official TOEFL guide. The primary objective is to assess the capabilities and constraints of ChatGPT, a prominent representative of large language models, within the context of automated essay scoring. The prevailing methodologies for automated essay scoring involve the utilization of deep neural networks, statistical machine learning techniques, and fine-tuning pre-trained models. However, these techniques face challenges when applied to different contexts or subjects, primarily due to their substantial data requirements and limited adaptability to small sample sizes. In contrast, this study employs ChatGPT to conduct an automated evaluation of English essays, even with a small sample size, employing an experimental approach. The empirical findings indicate that ChatGPT can provide operational functionality for automated essay scoring, although the results exhibit a regression effect. It is imperative to underscore that the effective design and implementation of ChatGPT prompts necessitate a profound domain expertise and technical proficiency, as these prompts are subject to specific threshold criteria. Keywords: ChatGPT, Automated Essay Scoring, Prompt Learning, TOEFL Independent Writing Task
Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring
National Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；School of Computer Science, Peking University（北京大学计算机学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）
Argument Rarity-based Originality Assessment for AI-Assisted Writing
Ritsumeikan Global Innovation Research Organization (R-GIRO), Ritsumeikan University（日光全球创新研究组织（R-GIRO）、日光大学）；Faculty of Business and Commerce, Kansai University（大阪大学商学部）；Arbege Corporation（Arbege公司）；College of Life Sciences, Ritsumeikan University（日光大学生命科学学院）；Office of General Education, Ritsumeikan University（日光大学通识教育办公室）
WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays
Department of Informatics, King’s College London（伦敦国王学院信息学系）；Waseda University（早稻田大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Sheffield（谢菲尔德大学）；Hosei University（法政大学）
From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（信息科技学院，科学大学，胡志明市，越南）；Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）
Chain-of-MetaWriting: Linguistic and Textual Analysis of How Small Language Models Write Young Students Texts
CNRS（法国国家科学研究中心）；University of Lorraine（洛林大学）；Sorbonne Nouvelle University（新索邦大学）；Institut Universitaire de France（法国大学研究院）；University of Strasbourg（斯特拉斯堡大学）
Chinese Essay Rhetoric Recognition Using LoRA, In-context Learning and Model Ensemble
The Open University of China（中国开放大学）；Engineering Research Center of Integration and Application of Digital Learning Technology, Ministry of Education（教育部数字学习技术整合与应用工程研究中心）
EssayJudge: A Multi-Granular Benchmark for Assessing Automated Essay Scoring Capabilities of Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing Future Brain Education Technology Co., Ltd.（北京未来脑教育科技有限公司）；The Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Guangxi Zhuang Autonomous Region Big Data Research Institute（广西壮族自治区大数据研究院）
How Well Does AI-Generated Feedback Work? Intrinsic and Extrinsic Evaluation across more than 20,000 EFL Essay Drafts
Tohoku University（东北大学）；RIKEN（理化学研究所）；ALTA Institute, Computer Laboratory, University of Cambridge（剑桥大学ALTA研究所，计算机实验室）；CNRS, LIS, Aix-Marseille University（法国国家科学研究中心，艾克斯-马赛大学语言信息处理实验室）；MBZUAI（Mohamed bin Zayed大学人工智能学院）
Language difference is one of the factors that hinder the acquisition of second language skills. In this article, we introduce a novel solution that leverages the strength of deep neural networks to measure the semantic dissimilarity between languages based on their word distributions in the embedding space of the multilingual pre-trained language model (e.g.,BERT). Then, we empirically examine the effectiveness of the proposed semantic language distance (SLD) in explaining the consistent variation in English ability of countries, which is proxied by their performance in the Internet-Based Tes…
This paper presents methods for improving automated essay scoring with techniques that address the computational trade-offs of self-attention and document length. To make Automated Essay Scoring (AES) more useful to practitioners, researchers must overcome the challenges of data and label availability, authentic and extended writing, domain scoring, prompt and source variety, and transfer learning. This paper addresses these challenges using neural network models by employing techniques that preserve essay length as an important feature without increasing model training costs. It introduces te…
Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs
Research Centre for Information Systems Engineering (LIRIS), KU Leuven（信息系统工程研究中心（LIRIS），鲁汶大学）；Department of Engineering Management, University of Antwerp（工程管理系，安特卫普大学）；Department of Decision Analytics and Risk, University of Southampton（决策分析与风险系，南安普顿大学）；School of Information, University of Michigan（信息学院，密歇根大学）
