Prompt Stability Scoring for Text Annotation with Large Language Models
Department of Sociology, New York University(纽约大学社会学系);Independent Researcher(独立研究者);Institute for Logic, Language, and Computation, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所)
Christopher Barrie, Elli Palaiologou, Petter Törnberg
Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package \texttt{promptstability} for its estimation. Using six different datasets and twelve outcomes, we classify $\sim$3.1m rows of data and $\sim$300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.
Composable Cross-prompt Essay Scoring by Merging Models
National Key Laboratory for Multimedia Information Processing(国家多媒体信息处理重点实验室);Peking University School of Computer Science(北京大学计算机学院);Peking University(北京大学)
Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring
National Key Laboratory for Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);School of Computer Science, Peking University(北京大学计算机学院);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院)
From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam(信息科技学院,科学大学,胡志明市,越南);Vietnam National University, Ho Chi Minh City, Vietnam(越南国家大学,胡志明市,越南)
iPOE: Interpretable Prompt Optimization via Explanations
Fundamentals of Natural Language Processing, University of Bamberg, Germany(自然语言处理基础,巴姆堡大学,德国);Leibniz-Institut für Psychologie (ZPID), Trier, Germany(莱比锡心理学研究所(ZPID),特里尔,德国)
Beyond Scores: A Modular RAG-Based System for Automatic Short Answer Scoring with Feedback
Graduate School of Information Science and Electrical Engineering(信息科学与电气工程研究生院);Kyushu University(九州大学);Faculty of Information Science and Electrical Engineering(信息科学与电气工程学部)
Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting
Institute of Artificial Intelligence and Future Networks, Beijing Normal University(北京师范大学人工智能与未来网络研究院);Hong Kong Baptist University(香港浸会大学);BNU-HKBU United International College(北京师范大学-香港浸会大学联合国际学院);College of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
Prompt Selection Matters: Enhancing Text Annotations for Social Sciences with Large Language Models
Université Paris 1 Panthéon-Sorbonne(巴黎第一大学先贤祠-索邦大学);Université Paris-Saclay(巴黎萨克雷大学);Institut Jean Nicod(让·尼科德研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL-EHESS-CNRS(巴黎文理研究大学-法国高等社会科学研究院-法国国家科学研究中心)
AI Rater Discrimination Depends on Scoring Protocol in Complex Clinical Decision-Making
Asclep Korea Inc.(Asclep韩国公司);Center for Data Science, New York University(纽约大学数据科学中心);Division of Endocrinology and Metabolism, Department of Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine(成均馆大学医学院内分泌与代谢科,三星医疗中心);Biomedical Statistics Center, Samsung Medical Center(三星医疗中心生物医学统计中心);Department of Digital Health, SAIHST(SAIHST数字健康科);Department of Data Convergence & Future Medicine, Sungkyunkwan University(成均馆大学数据融合与未来医学科)