Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Anthropic(Anthropic公司)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav … 展开作者
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan
We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University(国立台湾大学);Texas A&M University(德克萨斯A&M大学);Appier AI Research(Appier人工智能研究院);Graduate Institute of Communication Engineering, National Taiwan University(国立台湾大学通信工程研究所)
College of Computer Science, Sichuan University(四川大学计算机学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Leeds(利兹大学);The Alan Turing Institute(艾伦·图灵研究所)
AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligence
Seoul National University(首尔大学);Chung-Ang University(中央大学);NAVER AI Lab(NAVER AI实验室);NAVER Cloud(NAVER云);University of Richmond(里士满大学);Sogang University(西江大学)
Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior
Faculty of Computer Science, University of Vienna(计算机科学系,维也纳大学);UniVie Doctoral School Computer Science(UniVie 计算机科学博士学院);Faculty of Philological and Cultural Studies, University of Vienna(文学与文化研究系,维也纳大学)
When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);LG AI Research(LG人工智能研究院);University of Illinois Chicago(芝加哥大学伊利诺伊分校);University of Michigan(密歇根大学)
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
Department of Statistics, LSE(统计系,伦敦经济学院);Department of Mathematics, Tsinghua University(数学系,清华大学);School of Mathematics, University of Birmingham(数学学院,伯明翰大学);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Tencent Inc.(腾讯公司)
AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracy
London School of Economics and Political Science(伦敦政治经济学院);Massachusetts Institute of Technology(麻省理工学院);Federal Reserve Bank of Chicago(芝加哥联邦储备银行);University of California San Diego(加利福尼亚大学圣迭戈分校);University of Pennsylvania(宾夕法尼亚大学)
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
Georgia Institute of Technology(佐治亚理工学院);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学)