Recent breakthroughs in natural language processing (NLP) have permitted the synthesis and comprehension of coherent text in an open-ended way, therefore translating the theoretical algorithms into practical applications. The large language models (LLMs) have significantly impacted businesses such as report summarization software and copywriters. Observations indicate, however, that LLMs may exhibit social prejudice and toxicity, posing ethical and societal dangers of consequences resulting from irresponsibility. Large-scale benchmarks for accountable LLMs should consequently be developed. Although several empirical investigations reveal the existence of a few ethical difficulties in advanced LLMs, there is little systematic examination and user study of the risks and harmful behaviors of current LLM usage. To further educate future efforts on constructing ethical LLMs responsibly, we perform a qualitative research method called ``red teaming'' on OpenAI's ChatGPT\footnote{In this paper, ChatGPT refers to the version released on Dec 15th.} to better understand the practical features of ethical dangers in recent LLMs. We analyze ChatGPT comprehensively from four perspectives: 1) \textit{Bias} 2) \textit{Reliability} 3) \textit{Robustness} 4) \textit{Toxicity}. In accordance with our stated viewpoints, we empirically benchmark ChatGPT on multiple sample datasets. We find that a significant number of ethical risks cannot be addressed by existing benchmarks, and hence illustrate them via additional case studies. In addition, we examine the implications of our findings on AI ethics and harmal behaviors of ChatGPT, as well as future problems and practical design considerations for responsible LLMs. We believe that our findings may give light on future efforts to determine and mitigate the ethical hazards posed by machines in LLM applications.
Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking
Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, Valencia, Spain.(瓦伦西亚人工智能研究 institute,瓦伦西亚理工大学,西班牙瓦伦西亚);Department of Computer Science, The University of Chicago, Chicago, USA(计算机科学系,芝加哥大学,美国芝加哥);Center for Automation and Robotics, Spanish National Research Council, Madrid, Spain(自动化与机器人中心,西班牙国家研究委员会,西班牙马德里)
Red-TTT: Test-Time Training for Automated Jailbreaking Large Language Models
Johns Hopkins University(约翰斯·霍普金斯大学);National University of Singapore(新加坡国立大学);Washington University in St. Louis(圣路易斯华盛顿大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Stanford University(斯坦福大学)
Capability-Based Scaling Trends for LLM-Based Red-Teaming
ELLIS Institute Tübingen(图宾根ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Tübingen AI Center(图宾根人工智能中心);Foundation AI – Cisco Systems Inc.(AI基础研究机构——思科系统公司);EPFL(苏黎世联邦理工学院)
Learning diverse attacks on large language models for robust red-teaming and safety tuning
KAIST(韩国科学技术院);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);National University of Singapore(新加坡国立大学);University of Edinburgh(爱丁堡大学);CIFAR(加拿大高级研究所)
Can Small-Scale Data Poisoning Exacerbate Dialect-Linked Biases in Large Language Models?
Department of Electrical and Computer Engineering, Maroun Semaan Faculty of Engineering and Architecture(电气与计算机工程系,马鲁恩·塞马安工程与建筑学院);American University of Beirut(贝鲁特美国大学)
Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
Georgia Institute of Technology(佐治亚理工学院);University of Bristol(布里斯托大学);University of Maryland(马里兰大学);University College London(伦敦大学学院);MATS;Astra;MIT CSAIL(麻省理工学院计算机科学与人工智能实验室)
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Stanford University(斯坦福大学);University of California, Berkeley(加州大学伯克利分校);Center for AI Safety(AI安全中心);Microsoft Corporation(微软公司);The Chinese University of Hong Kong(香港中文大学)