Recent studies have demonstrated promising potential of ChatGPT for various text annotation and classification tasks. However, ChatGPT is non-deterministic which means that, as with human coders, identical input can lead to different outputs. Given this, it seems appropriate to test the reliability of ChatGPT. Therefore, this study investigates the consistency of ChatGPT's zero-shot capabilities for text annotation and classification, focusing on different model parameters, prompt variations, and repetitions of identical inputs. Based on the real-world classification task of differentiating website texts into news and not news, results show that consistency in ChatGPT's classification output can fall short of scientific thresholds for reliability. For example, even minor wording alterations in prompts or repeating the identical input can lead to varying outputs. Although pooling outputs from multiple repetitions can improve reliability, this study advises caution when using ChatGPT for zero-shot text annotation and underscores the need for thorough validation, such as comparison against human-annotated data. The unsupervised application of ChatGPT for text annotation and classification is not recommended.
Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳市大数据研究院);University of Science and Technology of China(中国科学技术大学)
Is ChatGPT a Good Personality Recognizer? A Preliminary Study
Institute of AI Education, East China Normal University(华东师范大学人工智能教育研究院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);School of Psychology and Cognitive Science, East China Normal University(华东师范大学心理与认知科学学院);Shanghai Changning Mental Health Center(上海市长宁区精神卫生中心)
Distinguishing Translations by Human, NMT, and ChatGPT: A Linguistic and Statistical Approach
School of Foreign Languages, Shanghai Jiao Tong University(上海交通大学外国语学院);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);Institute of Corpus Studies and Applications, Shanghai International Studies University(上海外国语大学语料库研究与应用研究院)
Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
Carnegie Mellon University(卡内基梅隆大学);LMU Munich(慕尼黑大学);University of Oxford(牛津大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Allen Institute for AI(艾伦人工智能研究所);IIT Delhi(德里印度理工学院)
Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis
College of Science and Engineering, Hamad Bin Khalifa University(哈马德·本·哈利法大学科学与工程学院);Department of Electronic and Computer Engineering, HKUST(香港科技大学电子与计算机工程系)
GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Engineering Research Center of Mixed Reality and Advanced Display(北京混合现实与先进显示工程研究中心);Institute of Technology, Beijing, China(北京理工大学)
Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions
University of Chicago(芝加哥大学);Syracuse University(雪城大学);New Jersey Institute of Technology(新泽西理工学院);Millburn High School(米尔本高中);WeChat AI(微信人工智能);Toyota Technological Institute at Chicago(芝加哥丰田技术学院)
DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4
The University of Georgia(佐治亚大学);Lehigh University(莱恩大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital and Harvard Medical School(麻省总医院和哈佛医学院);Mayo Clinic(梅奥诊所);ShanghaiTech University(上海科技大学);Shanghai United Imaging Intelligence Co., Ltd.(上海联影智能科技有限公司);Shanghai Clinical Research and Trial Center(上海临床研究与试验中心)