Discovering Language Model Behaviors with Model-Written Evaluations
Anthropic;Surge AI;Machine Intelligence Research Institute(机器智能研究所)
Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones,… 展开作者
Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan
As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.
Discovering Cyclists' Visual Preferences Through Shared Bike Trajectories and Street View Images Using Inverse Reinforcement Learning
Harbin Institute of Technology Shenzhen(哈尔滨工业大学(深圳));Shenzhen Technology University(深圳技术大学);Macau University of Science and Technology(澳门科技大学);Peking University(北京大学)
Discovering Differences in Strategic Behavior Between Humans and LLMs
Department of Computer Science, University of Texas at Austin. Work performed as a student researcher at Google DeepMind(德克萨斯大学计算机科学系。作为谷歌DeepMind的学生研究员进行的工作);Google DeepMind(谷歌DeepMind)
Discovering Phonetic Inventories with Crosslingual Automatic Speech Recognition
Center of Language and Speech Processing, The Johns Hopkins University(约翰斯·霍普金斯大学语言与语音处理中心);Human Language Technology Center of Excellence, The Johns Hopkins University(约翰斯·霍普金斯大学人类语言技术卓越中心);Multimedia Computing Group, Delft University of Technology(代尔夫特理工大学多媒体计算组);Department of Electrical and Computer Engineering, University of Illinois(伊利诺伊大学电气与计算机工程系)
Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models
Center for Language and Speech Processing (CLSP)(语言与语音处理中心);Johns Hopkins University(约翰霍普金斯大学);Group on Language, Audio & Music (GLAM)(语言、音频与音乐小组);Imperial College London(伦敦帝国学院)