arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.08073cs.CLcs.AI
收藏

宪法AI:来自AI反馈的无害性

Constitutional AI: Harmlessness from AI Feedback

Anthropic(Anthropic公司)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen… 展开作者

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan

摘要:

随着人工智能系统变得越来越强大,我们希望寻求他们的帮助来监督其他人工智能。我们尝试了通过自我改进来训练无害AI助手的方法,而没有任何人类标签来识别有害输出。唯一的人类监督是通过一系列规则或原则提供的,因此我们将这种方法称为“宪法AI”。该过程涉及监督学习和强化学习阶段。在监督阶段,我们从初始模型中进行采样,然后生成自我批评和修改,然后根据修改后的响应对原始模型进行微调。在RL阶段,我们从微调的模型中进行采样,使用模型来评估两个样本中哪一个更好,然后从这个AI偏好数据集训练偏好模型。然后,我们使用偏好模型作为奖励信号,使用RL进行训练,即我们使用“来自AI反馈的RL”(RLAIF)。因此,我们能够训练一个无害但不回避的人工智能助手,通过向他们解释其反对意见来处理有害的查询。SL和RL方法都可以利用思维链风格的推理来提高人工判断的性能和AI决策的透明度。这些方法可以更精确地控制人工智能的行为,并且使用更少的人类标签。

英文摘要:

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑