arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00215cs.SEcs.LG
收藏

LLM批评模型有助于发现LLM缺陷

LLM Critics Help Catch LLM Bugs

OpenAI

Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, Jan Leike

AI总结:

本文提出用RLHF训练的LLM批评模型辅助人类评估代码,实验表明其比人类评审更能发现LLM生成代码中的缺陷,并能识别ChatGPT训练数据中被标为无瑕疵的错误,人机协作还可减少幻觉。

摘要:

基于人类反馈的强化学习(RLHF)从根本上受限于人类正确评估模型输出的能力。为了提升人类评估能力并克服这一局限,本研究训练了“批评”模型,以帮助人类更准确地评估模型编写的代码。这些批评模型本身是经过RLHF训练的LLM,能够撰写自然语言反馈,指出真实世界助手任务中代码存在的问题。在包含自然产生的LLM错误的代码上,模型撰写的批评在63%的情况下比人类批评更受青睐,并且人类评估发现,模型比受雇进行代码审查的人类承包商能发现更多缺陷。我们进一步证实,经过微调的LLM批评模型能够成功识别出ChatGPT训练数据中被评定为“无瑕疵”的数百个错误,尽管这些任务大多数是非代码任务,因此对批评模型而言属于分布外样本。批评模型自身也可能存在局限,包括产生幻觉缺陷,这可能误导人类犯下原本可以避免的错误,但由批评模型与承包商组成的人机团队发现的缺陷数量与单独的LLM批评模型相近,同时比仅使用LLM时产生更少的幻觉。

英文摘要:

Reinforcement learning from human feedback (RLHF) is fundamentally limited by the capacity of humans to correctly evaluate model output. To improve human evaluation ability and overcome that limitation this work trains "critic" models that help humans to more accurately evaluate model-written code. These critics are themselves LLMs trained with RLHF to write natural language feedback highlighting problems in code from real-world assistant tasks. On code containing naturally occurring LLM errors model-written critiques are preferred over human critiques in 63% of cases, and human evaluation finds that models catch more bugs than human contractors paid for code review. We further confirm that our fine-tuned LLM critics can successfully identify hundreds of errors in ChatGPT training data rated as "flawless", even though the majority of those tasks are non-code tasks and thus out-of-distribution for the critic model. Critics can have limitations of their own, including hallucinated bugs that could mislead humans into making mistakes they might have otherwise avoided, but human-machine teams of critics and contractors catch similar numbers of bugs to LLM critics while hallucinating less than LLMs alone.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑