Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen… 展开作者
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.
Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);University of California, Berkeley(加州大学伯克利分校);Potsdam Institute for Climate Impact Research(波茨坦气候影响研究所);Ghent University(根特大学);Allen Institute for AI(艾伦人工智能研究所);University of Maryland(马里兰大学);EleutherAI;Union College(联合学院);Istanbul Bilgi University(伊斯坦布尔比尔吉大学)
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)(智能信息处理重点实验室,计算技术研究所,中国科学院);State Key Lab of AI Safety(人工智能安全国家重点实验室);University of Chinese Academy of Sciences, CAS(中国科学院大学);Zhejiang University(浙江大学)
Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment
School of Computer Science, Peking University(北京大学计算机学院);Key Laboratory of High Confidence Software Technologies(PKU), MOE, China(教育部高可信软件技术重点实验室(北京大学));Beijing Jiaotong University(北京交通大学)
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Qwen LLM Application Team, Alibaba(阿里巴巴Qwen大模型应用团队);Stanford University(斯坦福大学)