Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot -- i.e., without adaptation on downstream data. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations. However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot. In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.
ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory（中文信息处理实验室）；Ricoh Software Research Center (Beijing) Co., Ltd.（理光软件研究中心（北京）有限公司）
Is ChatGPT a Good Multi-Party Conversation Solver?
National Engineering Research Center of Speech and Language Information Processing（国家语音与语言信息处理工程技术研究中心）；University of Science and Technology of China（中国科学技术大学）
A Status Quo Investigation of Large Language Models towards Cost-Effective CFD Automation with OpenFOAMGPT: ChatGPT vs. Qwen vs. Deepseek
International Research Institute for Multidisciplinary Science, Beihang University, 100191 Beijing, China（多学科科学国际研究院，北航）；Cluster of Excellence SimTech, University of Stuttgart, Stuttgart, Germany（卓越仿真技术集群，斯图加特大学）；Faculty of Environment, Science and Economy, University of Exeter, Exeter EX4 4QF, United Kingdom（环境、科学与经济学院，埃克塞特大学）；Institute of Fluid Mechanics, Beihang University, 100191 Beijing, China（流体力学研究所，北航）；Institute of Thermodynamics（热力学研究所）
COMPARE: Clinical Optimization with Modular Planning and Assessment via RAG-Enhanced AI-OCT: Superior Decision Support for Percutaneous Coronary Intervention Compared to ChatGPT-5 and Junior Operators
Tangdu Hospital, The Fourth Military Medical University（空军军医大学唐都医院）；Research Institute of Tsinghua University in Shenzhen（深圳清华大学研究院）；The Second Affiliated Hospital, Air Force Medical University（空军军医大学第二附属医院）
Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
Carnegie Mellon University（卡内基梅隆大学）；LMU Munich（慕尼黑大学）；University of Oxford（牛津大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Allen Institute for AI（艾伦人工智能研究所）；IIT Delhi（德里印度理工学院）
Benchmarking ChatGPT-4 on ACR Radiation Oncology In-Training (TXIT) Exam and Red Journal Gray Zone Cases: Potentials and Challenges for AI-Assisted Medical Education and Decision Making in Radiation Oncology
University Hospital Erlangen（埃尔兰根大学医院）；Comprehensive Cancer Center Erlangen-EMN（埃尔兰根-EMN综合癌症中心）；Pattern Recognition Lab（模式识别实验室）
An Empirical Exploration of ChatGPT's Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability
University of Southern California（南加州大学）；Stevens Institute of Technology（史蒂文斯理工学院）；Virginia Tech（弗吉尼亚理工大学）
本文通过 NASA 太空任务设计挑战实证评估 ChatGPT-3.5 在任务工程问题表述中识别利益相关者的能力，发现其对人本利益相关者识别较好但对环境因素识别差、抽象层次保持不足且并行输出变异性大，表明其可靠性有限。
General-purpose AI models can generate actionable knowledge on agroecological crop protection
Chrysalis Consulting（Chrysalis咨询公司）；Institute for Plant Protection, China Academy of Agricultural Sciences（中国农业科学院植物保护研究所）；School of the Environment, University of Queensland（昆士兰大学环境学院）
InstaSynth: Opportunities and Challenges in Generating Synthetic Instagram Data with ChatGPT for Sponsored Content Detection
Utrecht University（乌得勒支大学）；University College Maastricht, Maastricht University（马斯特里赫特大学马斯特里赫特大学学院）；Indira Gandhi Delhi Technical University for Women（英迪拉·甘地德里女子技术大学）
Leveraging ChatGPT As Text Annotation Tool For Sentiment Analysis
College of Science and Engineering, Hamad Bin Khalifa University（哈马德·本·哈利法大学科学与工程学院）；Department of Electronic and Computer Engineering, HKUST（香港科技大学电子与计算机工程系）
