arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.07858cs.CLcs.AIcs.CY
收藏

减少伤害的红队测试语言模型:方法、缩放行为和经验教训

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Anthropic(Anthropic公司)
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sa… 展开作者

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark

摘要:

我们描述了我们早期的努力,红队语言模型,以同时发现,测量,并试图减少其潜在的有害输出。我们做出了三个主要贡献。首先,我们研究了红队测试在3种模型大小(2.7 B、13 B和52 B参数)和4种模型类型下的缩放行为:普通语言模型(LM);提示有用、诚实和无害的LM;具有拒绝采样的LM;以及使用基于人类反馈的强化学习(RLHF)训练为有用和无害的模型。我们发现,RLHF模型越来越难以红队,因为他们的规模,我们发现一个平坦的趋势与规模的其他模型类型。其次,我们发布了38,961次红队攻击的数据集,供其他人分析和学习。我们提供了自己的数据分析,并发现了各种有害的输出,从攻击性语言到更微妙的有害的非暴力不道德输出。第三,我们详尽地描述了我们的指示,流程,统计方法,以及关于红队测试的不确定性。我们希望这种透明性能够加速我们作为一个社区共同工作的能力,以便为如何红色团队语言模型开发共享的规范、实践和技术标准。

英文摘要:

We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs. We make three main contributions. First, we investigate scaling behaviors for red teaming across 3 model sizes (2.7B, 13B, and 52B parameters) and 4 model types: a plain language model (LM); an LM prompted to be helpful, honest, and harmless; an LM with rejection sampling; and a model trained to be helpful and harmless using reinforcement learning from human feedback (RLHF). We find that the RLHF models are increasingly difficult to red team as they scale, and we find a flat trend with scale for the other model types. Second, we release our dataset of 38,961 red team attacks for others to analyze and learn from. We provide our own analysis of the data and find a variety of harmful outputs, which range from offensive language to more subtly harmful non-violent unethical outputs. Third, we exhaustively describe our instructions, processes, statistical methodologies, and uncertainty about red teaming. We hope that this transparency accelerates our ability to work together as a community in order to develop shared norms, practices, and technical standards for how to red team language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑