arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.02039cs.CL
收藏

基于大语言模型的文本标注提示稳定性评分

Prompt Stability Scoring for Text Annotation with Large Language Models

Department of Sociology, New York University(纽约大学社会学系);Independent Researcher(独立研究者);Institute for Logic, Language, and Computation, University of Amsterdam(阿姆斯特丹大学逻辑、语言与计算研究所)

Christopher Barrie, Elli Palaiologou, Petter Törnberg

AI总结:

本文提出一种通用框架,通过传统方法改进内在和跨编码器可靠性评分,提出提示稳定性评分(PSS)及Python包promptstability,用于诊断低稳定性提示并展示功能。

摘要:

研究人员越来越多地使用语言模型(LMs)进行文本标注。这些方法仅依赖提示告诉模型根据指令返回给定输出。然而,LM输出的可重复性可能对提示设计的微小变化敏感,这质疑了分类流程的可重复性。为解决此问题,研究者通常测试多种语义相似的提示以确定所谓的

英文摘要:

Researchers are increasingly using language models (LMs) for text annotation. These approaches rely only on a prompt telling the model to return a given output according to a set of instructions. The reproducibility of LM outputs may nonetheless be vulnerable to small changes in the prompt design. This calls into question the replicability of classification routines. To tackle this problem, researchers have typically tested a variety of semantically similar prompts to determine what we call ``prompt stability." These approaches remain ad-hoc and task specific. In this article, we propose a general framework for diagnosing prompt stability by adapting traditional approaches to intra- and inter-coder reliability scoring. We call the resulting metric the Prompt Stability Score (PSS) and provide a Python package \texttt{promptstability} for its estimation. Using six different datasets and twelve outcomes, we classify $\sim$3.1m rows of data and $\sim$300m input tokens to: a) diagnose when prompt stability is low; and b) demonstrate the functionality of the package. We conclude by providing best practice recommendations for applied researchers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑