arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.09251cs.CLcs.AIcs.LG
收藏

使用模型编写的求值发现语言模型行为

Discovering Language Model Behaviors with Model-Written Evaluations

Anthropic;Surge AI;Machine Intelligence Research Institute(机器智能研究所)
Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones,… 展开作者

Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan

摘要:

随着语言模型(LM)的扩展,它们会开发出许多新的行为,有好有坏,这加剧了评估它们行为方式的需求。先前的工作使用众包(耗时且昂贵)或现有数据源(并不总是可用)创建评估。在这里,我们使用LM自动生成评估。我们探索的方法与不同数量的人力,从指示LM写是/否的问题,使复杂的Winogender模式与多个阶段的LM为基础的生成和过滤。Crowdworker将这些示例评为高度相关,并与90-100%的标签一致,有时甚至比相应的人类书面数据集更一致。我们生成了154个数据集,并发现了逆缩放的新情况,其中LM随着大小而变差。较大的LM重复回对话用户的首选答案(“谄媚”),并表达更大的愿望,追求有关的目标,如资源获取和目标保存。我们还发现了一些第一个例子的逆缩放RL从人的反馈(RLHF),更多的RLHF使LM更糟。例如,RLHF使LM表达更强烈的政治观点(关于枪支权利和移民)和更大的避免关闭的愿望。总的来说,LM撰写的评价是高质量的,让我们很快发现了许多新颖的LM行为。

英文摘要:

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑