LLMs can Perform Multi-Dimensional Analytic Writing Assessments: A Case Study of L2 Graduate-Level Academic English Writing
Department of Linguistics & Institute for Advanced Computational Science, Stony Brook University（语言学系与高级计算科学研究所，石溪大学）；Department of Linguistics, University of Saskatchewan（语言学系，萨斯喀彻温大学）
Zhengxiang Wang, Veronika Makarova, Zhi Li, Jordan Kodner, Owen Rambow
The paper explores the performance of LLMs in the context of multi-dimensional analytic writing assessments, i.e. their ability to provide both scores and comments based on multiple assessment criteria. Using a corpus of literature reviews written by L2 graduate students and assessed by human experts against 9 analytic criteria, we prompt several popular LLMs to perform the same task under various conditions. To evaluate the quality of feedback comments, we apply a novel feedback comment quality evaluation framework. This framework is interpretable, cost-efficient, scalable, and reproducible, compared to existing methods that rely on manual judgments. We find that LLMs can generate reasonably good and generally reliable multi-dimensional analytic assessments. We release our corpus and code for reproducibility.
Meow: End-to-End Outline Writing for Automatic Academic Survey
Beijing Wenge Technology Co.,Ltd（北京文格科技有限公司）；Peking University（北京大学）；Duke University（杜克大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
OverleafCopilot: Empowering Academic Writing in Overleaf with Large Language Models
School of Computer and Information Technology, Beijing Jiaotong University（北京交通大学计算机与信息技术学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Southern University of Science and Technology（南方科技大学）；National University of Singapore (Chongqing Research Institute)（新加坡国立大学（重庆研究机构））；Fudan University（复旦大学）；Qilu University of Technology（齐鲁大学）；CISPA Helmholtz Center for Information Security（CISPA 欧洲信息安全研究所）
CoachGPT: A Scaffolding-based Academic Writing Assistant
Institute for Financial Services Analytics（金融服务分析研究所）；University of Delaware（德克萨斯大学）；School of Education（教育学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）
Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt（普遍知识处理实验室（UKP实验室）、计算机科学系和海斯曼人工智能中心（hessian.AI）、达姆施塔特技术大学）；Department of Computer Science at Queens College, City University of New York (CUNY)（纽约城市大学皇后学院计算机科学系）
Measuring Human Involvement in AI-Generated Text: A Case Study on Academic Writing
Dept. of Information & Communication Engineering (ICE), The University of Tokyo (Japan)（信息与通信工程系（ICE），东京大学（日本））；National Institute of Informatics (Japan)（日本信息处理学会）
Large Language Models Penetration in Scholarly Writing and Peer Review
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Electronic Science and Technology of China（电子科技大学）；Wuhan University of Technology（武汉理工大学）；University of Copenhagen（哥本哈根大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）
EduResearchBench: A Hierarchical Atomic Task Decomposition Benchmark for Full-Lifecycle Educational Research
Shanghai Institute of AI for Education, East China Normal University（上海人工智能教育研究院，东华大学）；Shanghai Innovation Institute（上海创新研究院）；School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）
University of Portsmouth（朴茨茅斯大学）；Varna University of Management（瓦尔纳管理大学）；Zangador Research Institute（赞加多尔研究所）；Bournemouth University（伯恩茅斯大学）
The paper looks at the role of large language models in academic knowledge creation based on a scoping review (2018 to January 2023) of how researchers have previously used the language model GPT to assist in the performance of academic knowledge creation tasks beyond data analysis. These tasks include writing, editing, reviewing, dataset creation and curation, which have been difficult to perform using earlier ML tools. Based on a synthesis of these papers, this study identifies pathways for a future academic research landscape that incorporates wider usage of large language models based on t…
Inclusivity in Large Language Models: Personality Traits and Gender Bias in Scientific Abstracts
Information Sciences Institute, University of Southern California（南加州大学信息科学学院）；Iovine and Young Academy, University of Southern California（南加州大学约万与杨学院）；In Vivo Group
From Intention to Text: AI-Supported Goal Setting in Academic Writing
Department of Media Technology and Interaction Design（媒体技术与交互设计系）；KTH Royal Institute of Technology（皇家理工学院）；Department of Digital Learning（数字学习系）；Digital Futures（数字未来）
Universität Hamburg（汉堡大学）；International Institute of Information Technology, Bangalore（班加罗尔国际信息技术学院）；City University of Hong Kong（香港城市大学）
We present the first approach to automatically building resources for academic writing. The aim is to build a writing aid system that automatically edits a text so that it better adheres to the academic style of writing. On top of existing academic resources, such as the Corpus of Contemporary American English (COCA) academic Word List, the New Academic Word List, and the Academic Collocation List, we also explore how to dynamically build such resources that would be used to automatically identify informal or non-academic words or phrases. The resources are compiled using different generic app…
Delving into LLM-assisted writing in biomedical publications through excess vocabulary
Hertie Institute for AI in Brain Health, University of Tübingen, Germany（人工智能与脑健康赫尔蒂研究所、图宾根大学，德国）；Northwestern University, Evanston, Illinois, USA（西北大学，埃文斯顿，伊利诺伊州，美国）
Tohoku University（东北大学）；Langsmith Inc.（Langsmith公司）；Yahoo Japan Corporation（雅虎日本公司）；RIKEN（理化学研究所）；Octanove Labs LLC（Octanove实验室有限责任公司）
The writing process consists of several stages such as drafting, revising, editing, and proofreading. Studies on writing assistance, such as grammatical error correction (GEC), have mainly focused on sentence editing and proofreading, where surface-level issues such as typographical, spelling, or grammatical errors should be corrected. We broaden this focus to include the earlier revising stage, where sentences require adjustment to the information included or major rewriting and propose Sentence-level Revision (SentRev) as a new writing assistance task. Well-performing systems in this task ca…
The strength with which a statement is made can have a significant impact on the audience. For example, international relations can be strained by how the media in one country describes an event in another; and papers can be rejected because they overstate or understate their findings. It is thus important to understand the effects of statement strength. A first step is to be able to distinguish between strong and weak statements. However, even this problem is understudied, partly due to a lack of data. Since strength is inherently relative, revisions of texts that make claims are a natural so…
National Research & Engineering Center for E-learning（国家数字化学习工程技术研究中心）；Key Lab. of Adolescent Cyberpsychology and Behavior, Ministry of Education（教育部青少年网络心理与行为重点实验室）；University of Memphis（孟菲斯大学）；Institute for Intelligent Systems, University of Memphis（孟菲斯大学智能系统研究所）；Tsinghua University（清华大学）
The relationship between reading and writing (RRW) is one of the major themes in learning science. One of its obstacles is that it is difficult to define or measure the latent background knowledge of the individual. However, in an academic research setting, scholars are required to explicitly list their background knowledge in the citation sections of their manuscripts. This unique opportunity was taken advantage of to observe RRW, especially in the published academic commentary scenario. RRW was visualized under a proposed topic process model by using a state of the art version of latent Diri…
ChatGPT has enabled access to AI-generated writing for the masses, and within just a few months, this product has disrupted the knowledge economy, initiating a culture shift in the way people work, learn, and write. The need to discriminate human writing from AI is now both critical and urgent, particularly in domains like higher education and academic writing, where AI had not been a significant threat or contributor to authorship. Addressing this need, we developed a method for discriminating text generated by ChatGPT from (human) academic scientists, relying on prevalent and accessible supe…
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（矢量人工智能研究院）；Li Ka Shing Knowledge Institute, Unity Health Toronto（多伦多统一健康李嘉诚知识研究所）；Surgical Safety Technologies（手术安全技术公司）
As the numbers of submissions to conferences grow quickly, the task of assessing the quality of academic papers automatically, convincingly, and with high accuracy attracts increasing attention. We argue that studying interpretable dimensions of these submissions could lead to scalable solutions. We extract a collection of writing features, and construct a suite of prediction tasks to assess the usefulness of these features in predicting citation counts and the publication of AI-related papers. Depending on the venues, the writing features can predict the conference vs. workshop appearance wit…
Measuring How Students Rely on Generative AI in Academic Writing: Development and Multi-Source Validation of the Generative AI Reliance Types Scale (GenAI-RTS)
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）；University at Buffalo（布法罗大学）
