ChatGPT has become a global sensation. As ChatGPT and other Large Language Models (LLMs) emerge, concerns of misusing them in various ways increase, such as disseminating fake news, plagiarism, manipulating public opinion, cheating, and fraud. Hence, distinguishing AI-generated from human-generated becomes increasingly essential. Researchers have proposed various detection methodologies, ranging from basic binary classifiers to more complex deep-learning models. Some detection techniques rely on statistical characteristics or syntactic patterns, while others incorporate semantic or contextual information to improve accuracy. The primary objective of this study is to provide a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. Additionally, we evaluated other AI-generated text detection tools that do not specifically claim to detect ChatGPT-generated content to assess their performance in detecting ChatGPT-generated content. For our evaluation, we have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains and user-generated responses from popular social networking platforms. The dataset serves as a reference to assess the performance of various techniques in detecting ChatGPT-generated content. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory(中文信息处理实验室);Ricoh Software Research Center (Beijing) Co., Ltd.(理光软件研究中心(北京)有限公司)
Investigating the Effectiveness of ChatGPT in Mathematical Reasoning and Problem Solving: Evidence from the Vietnamese National High School Graduation Examination
Eastern International University(东方国际大学);International University, VNUHCM(越南国立大学胡志明市国际大学)
University of California, Los Angeles(加州大学洛杉矶分校);Nanyang Technological University(南洋理工大学);University of California, Davis(加州大学戴维斯分校);National University of Singapore(新加坡国立大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education
University of Massachusetts Boston(马萨诸塞大学波士顿分校);Texas Tech University(德克萨斯理工大学);Coventry University(考文垂大学);MacNeal Hospital(麦克尼尔医院);Washington State University(华盛顿州立大学)
本研究评估了 ChatGPT 回答复杂医学临床问题的可靠性,发现其在 USMLE 等测试中表现出较好的演绎推理能力,伦理问题得分达 60%,逻辑问题达 58.8%,证明其可作为 AI 辅助医学教育的潜力工具,但准确性仍待提升。
Performance of ChatGPT-3.5 and GPT-4 on the United States Medical Licensing Examination With and Without Distractions
Magee-Womens Research Institute(马吉妇女研究所);University of Pittsburgh School of Medicine(匹兹堡大学医学院);Chaim Sheba Medical Center(哈伊姆·谢巴医疗中心);Sackler Faculty of Medicine, Tel Aviv University(特拉维夫大学萨克勒医学院);Ariel University(阿里尔大学)
ChatGPT for automated grading of short answer questions in mechanical ventilation
Shri Atal Bihari Vajpayee Medical College and Research Institute(阿塔尔·比哈里·瓦杰帕伊医学院与研究所);Bowring and Lady Curzon Hospitals(鲍林与莱迪·柯曾医院);Rajiv Gandhi University of Health Sciences(拉吉夫·甘地健康科学大学);Intensive Care Service, Westmead Hospital(韦斯特米德医院重症监护服务部)
The Application of ChatGPT in Responding to Questions Related to the Boston Bowel Preparation Scale
First Hospital of Quanzhou Affiliated to Fujian Medical University(福建医科大学附属泉州第一医院);McGill University(麦吉尔大学);Montreal Neurological Institute(蒙特利尔神经学研究所);Mila - Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所)
Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions
University of Chicago(芝加哥大学);Syracuse University(雪城大学);New Jersey Institute of Technology(新泽西理工学院);Millburn High School(米尔本高中);WeChat AI(微信人工智能);Toyota Technological Institute at Chicago(芝加哥丰田技术学院)
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
York University(约克大学);Nanyang Technological University(南洋理工大学);Dialpad Canada Inc.(Dialpad加拿大公司);Royal Bank of Canada(加拿大皇家银行);Salesforce Research(赛富时研究院)
Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Virginia Tech(弗吉尼亚理工大学);Huazhong University of Science and Technology(华中科技大学)
ChatGPT and Bard Responses to Polarizing Questions
Missouri S & T(密苏里科技大学);Yale University(耶鲁大学);Jivox(Jivox公司);Limbik(Limbik公司);Memorial Sloan Kettering Cancer Center(纪念斯隆凯特琳癌症中心);Georgia Tech University(佐治亚理工学院);Singapore University of Technology(新加坡科技大学);New York University(纽约大学)
ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports
University Hospital, LMU Munich(慕尼黑大学附属医院);LMU Munich(慕尼黑大学);Comprehensive Pneumology Center (CPC-M)(综合肺病中心(CPC-M));German Center for Lung Research (DZL)(德国肺研究中心(DZL));Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心(MCML))
Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
Mass General Brigham, Harvard Medical School(麻省总医院布莱根,哈佛医学院);Brigham and Women’s Hospital/Dana-Farber Cancer Institute(布莱根妇女医院/丹娜-法伯癌症研究所);Technische Universität Darmstadt(达姆施塔特工业大学);Maastricht University(马斯特里赫特大学)
HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Authority Signals in AI Cited Health Sources: A Framework for Evaluating Source Credibility in ChatGPT Responses
York College, The City University of New York(纽约市立大学约克学院);Teachers College, Columbia University(哥伦比亚大学教师学院);William Paterson University(威廉帕特森大学);CUNY School of Public Health(纽约市立大学公共卫生学院)