Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe responses. Red teaming is instrumental for both model alignment and evaluation, but is labor-intensive and difficult to scale when done by humans. In this paper, we present Gradient-Based Red Teaming (GBRT), a red teaming method for automatically generating diverse prompts that are likely to cause an LM to output unsafe responses. GBRT is a form of prompt learning, trained by scoring an LM response with a safety classifier and then backpropagating through the frozen safety classifier and LM to update the prompt. To improve the coherence of input prompts, we introduce two variants that add a realism loss and fine-tune a pretrained model to generate the prompts instead of learning the prompts directly. Our experiments show that GBRT is more effective at finding prompts that trigger an LM to generate unsafe responses than a strong reinforcement learning-based red teaming approach, and succeeds even when the LM has been fine-tuned to produce safer outputs.
Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
LibrAI;Tsinghua University(清华大学);Harbin Institute of Technology(哈尔滨工业大学);Monash University(莫纳什大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);The University of Melbourne(墨尔本大学)
Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents
School of Software, Henan University(河南大学软件学院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学)
Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
Responsible Tech Research IBM Research Yorktown Heights New York USA(负责技术研究 IBM 研究 Yorktown Heights 新 York 美国);IBM Research(IBM 研究);The Pennsylvania State University(宾夕法尼亚州立大学);University of Notre Dame(诺特达美大学)
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Center for AI Safety(人工智能安全中心);Carnegie Mellon University(卡内基梅隆大学);UC Berkeley(加州大学伯克利分校);Microsoft(微软)
Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
Fudan University(复旦大学);City University of Hong Kong(香港城市大学);Deakin University(德克萨斯大学);Hon Hai Research Institute(鸿海研究室);Hong Kong University of Science and Technology(香港理工大学)
Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming
Paul G. Allen School of Computer Science & Engineering(保罗·G·艾伦计算机科学与工程学院);IT University of Copenhagen(哥本哈根信息技术大学);Berkeley Center for Human-Compatible AI, UC Berkeley(加州大学伯克利分校伯克利人类兼容人工智能中心);NVIDIA Corporation(英伟达公司)
University of Toronto(多伦多大学);Vector Institute for AI(人工智能向量研究所);Johns Hopkins Medical Institutions(约翰霍普金斯医学机构);University Health Network(医疗网络);Algoma University(阿尔戈马大学);University of Iowa Hospitals & Clinics(爱荷华大学医院与诊所);University of Edinburgh(爱丁堡大学)
Learning diverse attacks on large language models for robust red-teaming and safety tuning
KAIST(韩国科学技术院);Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);National University of Singapore(新加坡国立大学);University of Edinburgh(爱丁堡大学);CIFAR(加拿大高级研究所)
Resource Consumption Red-Teaming for Large Vision-Language Models
China Mobile Research Institute(中国移动研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Nanyang Technological University(南洋理工大学);University of Science and Technology of China(中国科学技术大学);North China Electric Power University(华北电力大学)
Prompt Optimization and Evaluation for LLM Automated Red Teaming
Fuel iX Applied Research(Fuel iX应用研究);North Carolina State University(北卡罗来纳州立大学);University of Minnesota(明尼苏达大学);TELUS Digital(TELUS数字);Learn Prompting
RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室);School of Cyber Science and Technology, Zhejiang University(浙江大学网络科学与技术学院);Palo Alto Networks(派拓网络);University of North Texas(北得克萨斯大学)
Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
Independent Researcher(独立研究者);Hunan University(湖南大学);Shenzhen Kaihong Digital Industry Development Co., Ltd.(深圳凯鸿数字产业开发有限公司);Central University of Finance and Economics(中央财经大学);Chongqing University(重庆大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Stevens Institute of Technology(斯蒂文斯理工学院);Cornell University(康奈尔大学);Oak Ridge National Laboratory(橡树岭国家实验室);Zhengzhou University of Light Industry(郑州轻工业大学);Xidian University(西安电子科技大学);The University of Texas at Dallas(德克萨斯大学达拉斯分校);AI Safety Research Lab, Institute of Advanced Computing(高级计算研究所人工智能安全研究实验室);University of Malaya(马来亚大学);Emory University(埃默里大学);Department of Nephrology, Affiliated Hospital of Guangdong Medical University(广东医学院附属医院肾内科)