arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.11085cs.CL
收藏

检验 ChatGPT 在文本标注与分类中的可靠性:一项警示性说明

Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark

University of Zurich(苏黎世大学)

Michael V. Reiss

AI总结:

本文通过新闻/非新闻分类任务测试 ChatGPT 零样本文本标注的一致性,发现提示词微调或重复输入即可改变结果,提示应充分验证且不建议无监督使用。

摘要:

近期研究表明,ChatGPT 在各类文本标注和分类任务中具有可观潜力。然而,ChatGPT 是非确定性的,这意味着与人工编码员一样,相同输入也可能产生不同输出。鉴于此,检验 ChatGPT 的可靠性显得十分必要。因此,本研究考察 ChatGPT 在文本标注与分类中零样本能力的一致性,重点关注不同模型参数、提示词变化以及相同输入的重复运行。基于将网站文本区分为新闻与非新闻的真实分类任务,结果显示,ChatGPT 分类输出的一致性可能低于可靠性的科学阈值。例如,即使提示词中细微的措辞改动,或重复提交相同输入,也会导致输出发生变化。尽管汇总多次重复的输出可以提高可靠性,但本研究建议在将 ChatGPT 用于零样本文本标注时保持谨慎,并强调需要进行充分验证,例如与人工标注数据进行比较。不建议以无监督方式应用 ChatGPT 进行文本标注和分类。

英文摘要:

Recent studies have demonstrated promising potential of ChatGPT for various text annotation and classification tasks. However, ChatGPT is non-deterministic which means that, as with human coders, identical input can lead to different outputs. Given this, it seems appropriate to test the reliability of ChatGPT. Therefore, this study investigates the consistency of ChatGPT's zero-shot capabilities for text annotation and classification, focusing on different model parameters, prompt variations, and repetitions of identical inputs. Based on the real-world classification task of differentiating website texts into news and not news, results show that consistency in ChatGPT's classification output can fall short of scientific thresholds for reliability. For example, even minor wording alterations in prompts or repeating the identical input can lead to varying outputs. Although pooling outputs from multiple repetitions can improve reliability, this study advises caution when using ChatGPT for zero-shot text annotation and underscores the need for thorough validation, such as comparison against human-annotated data. The unsupervised application of ChatGPT for text annotation and classification is not recommended.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑