Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Anthropic(Anthropic公司)
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sa… 展开作者
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark
We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Center for AI Safety(人工智能安全中心);Carnegie Mellon University(卡内基梅隆大学);UC Berkeley(加州大学伯克利分校);Microsoft(微软)
Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation
Responsible Tech Research IBM Research Yorktown Heights New York USA(负责技术研究 IBM 研究 Yorktown Heights 新 York 美国);IBM Research(IBM 研究);The Pennsylvania State University(宾夕法尼亚州立大学);University of Notre Dame(诺特达美大学)
Ask What Your Country Can Do For You: Towards a Public Red Teaming Model
Oxford Internet Institute(牛津互联网研究所);University of Oxford(牛津大学);Amazon(亚马逊);Department of Computer Science(计算机科学系);University of Wisconsin - Eau Claire(威斯康星大学欧克莱尔分校);Carnegie Mellon University(卡内基梅隆大学);Harvard University(哈佛大学)
Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation
Harvard University(哈佛大学);Google Research(谷歌研究院);University of Zurich(苏黎世大学);Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);University College London(伦敦大学学院);Cohere(科here公司);MLCommons(MLCommons组织)
Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South
Google DeepMind(谷歌DeepMind);Yonsei University(延世大学);Imperial College(帝国理工学院);University of Wisconsin–Stevens Point(威斯康星州立大学斯普林特分校);Google Research(谷歌研究);Harvard University(哈佛大学)
Red-Teaming Text-to-Image Systems by Rule-based Preference Modeling
KLMM, UCAS, Academy of Mathematics and Systems Science, Chinese Academy of Sciences(UCAS信息科学学院、数学与系统科学研究院、中国科学院);College of AI, Tsinghua University(清华大学人工智能学院)
REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming
King Fahd University of Petroleum & Minerals(法赫德国王石油与矿产大学);Imam Abdulrahman Bin Faisal University(伊玛目阿卜杜勒拉赫曼·本·费萨尔大学);SDAIA-KFUPM Joint Research Center for Artificial Intelligence(SDAIA-KFUPM人工智能联合研究中心)
DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling
School of Cyber Science(网络安全学院);State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Tsinghua University(清华大学)
A Mechanism-Based Approach to Mitigating Harms from Persuasive Generative AI
Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Jigsaw(拼图工作室);University of Edinburgh(爱丁堡大学);Cornell University(康奈尔大学);University College London(伦敦大学学院)
RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
Fudan University(复旦大学);City University of Hong Kong(香港城市大学);Deakin University(德克萨斯大学);Hon Hai Research Institute(鸿海研究室);Hong Kong University of Science and Technology(香港理工大学)