Learning diverse attacks on large language models for robust red-teaming and safety tuning
KAIST（韩国科学技术院）；Mila – Québec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；National University of Singapore（新加坡国立大学）；University of Edinburgh（爱丁堡大学）；CIFAR（加拿大高级研究所）
Seanie Lee, Minsu Kim, Lynn Cherif, David Dobre, Juho Lee, Sung Ju Hwang, Kenji Kawaguchi, Gauthier Gidel, Yoshua Bengio, Esmeralda S. Whitammer, Moksh Jain
Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe and responsible deployment of large language models (LLMs). Developing effective protection against many modes of attack prompts requires discovering diverse attacks. Automated red-teaming typically uses reinforcement learning to fine-tune an attacker language model to generate prompts that elicit undesirable responses from a target LLM, as measured, for example, by an auxiliary toxicity classifier. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. As a flexible and probabilistically principled alternative, we propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts. We find that the attacks generated by our method are effective against a wide range of target LLMs, both with and without safety tuning, and transfer well between target LLMs. Finally, we demonstrate that models safety-tuned using a dataset of red-teaming prompts generated by our method are robust to attacks from other RL-based red-teaming approaches.
TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesis
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University（航空信息安全与可信计算重点实验室，教育部，网络安全工程学院，武汉大学）；Ant Group（蚂蚁集团）；Ant International（蚂蚁国际）；School of Computing Technologies, Royal Melbourne Institute of Technology（计算技术学院，皇家墨尔本理工学院）
Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
Independent Researcher（独立研究者）；Hunan University（湖南大学）；Shenzhen Kaihong Digital Industry Development Co., Ltd.（深圳凯鸿数字产业开发有限公司）；Central University of Finance and Economics（中央财经大学）；Chongqing University（重庆大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Stevens Institute of Technology（斯蒂文斯理工学院）；Cornell University（康奈尔大学）；Oak Ridge National Laboratory（橡树岭国家实验室）；Zhengzhou University of Light Industry（郑州轻工业大学）；Xidian University（西安电子科技大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）；AI Safety Research Lab, Institute of Advanced Computing（高级计算研究所人工智能安全研究实验室）；University of Malaya（马来亚大学）；Emory University（埃默里大学）；Department of Nephrology, Affiliated Hospital of Guangdong Medical University（广东医学院附属医院肾内科）
Capability-Based Scaling Trends for LLM-Based Red-Teaming
ELLIS Institute Tübingen（图宾根ELLIS研究所）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；Tübingen AI Center（图宾根人工智能中心）；Foundation AI – Cisco Systems Inc.（AI基础研究机构——思科系统公司）；EPFL（苏黎世联邦理工学院）
Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
Responsible Tech Research IBM Research Yorktown Heights New York USA（负责技术研究 IBM 研究 Yorktown Heights 新 York 美国）；IBM Research（IBM 研究）；The Pennsylvania State University（宾夕法尼亚州立大学）；University of Notre Dame（诺特达美大学）
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation
King Fahd University of Petroleum & Minerals（法赫德国王石油矿产大学）；Imam Abdulrahman Bin Faisal University（伊玛目阿卜杜勒拉赫曼·本·费萨尔大学）；SDAIA-KFUPM Joint Research Center for Artificial Intelligence（沙特数据与人工智能局-法赫德国王石油矿产大学人工智能联合研究中心）
RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）；Deakin University（德克萨斯大学）；Hon Hai Research Institute（鸿海研究室）；Hong Kong University of Science and Technology（香港理工大学）
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Center for AI Safety（人工智能安全中心）；Carnegie Mellon University（卡内基梅隆大学）；UC Berkeley（加州大学伯克利分校）；Microsoft（微软）
Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming
Paul G. Allen School of Computer Science & Engineering（保罗·G·艾伦计算机科学与工程学院）；IT University of Copenhagen（哥本哈根信息技术大学）；Berkeley Center for Human-Compatible AI, UC Berkeley（加州大学伯克利分校伯克利人类兼容人工智能中心）；NVIDIA Corporation（英伟达公司）
