arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.18219cs.LGcs.AIcs.CL
收藏

递归内省:教语言模型智能体如何自我改进

Recursive Introspection: Teaching Language Model Agents How to Self-Improve

Carnegie Mellon University(卡内基梅隆大学);UC Berkeley(加州大学伯克利分校);MultiOn

Yuxiao Qu, Tianjun Zhang, Naman Garg, Aviral Kumar

AI总结:

本文提出 RISE 方法,通过将单轮提示微调转化为多轮马尔可夫决策过程并进行迭代微调,使大语言模型能够递归地检测和纠正先前错误,在数学推理任务上实现自我改进且不破坏单轮能力。

摘要:

在基础模型中实现智能体行为的核心之一,是使其能够在有更多计算或交互时,对自身行为进行内省、推理并纠正错误。即使是最强大的专有大语言模型(LLMs),也并未完全展现出在序列上持续改进其回复的能力,即使在被明确告知犯错时也是如此。在本文中,我们开发了 RISE:递归内省,一种通过微调 LLMs 来引入这一能力的方法,尽管先前的工作假设这种能力可能无法实现。我们的方法规定了一个迭代微调过程,试图教模型在执行先前解决困难测试时问题的失败尝试后,如何改变其回复,并可选地利用额外的环境反馈。RISE 将单轮提示的微调视为求解一个多轮马尔可夫决策过程(MDP),其初始状态为提示。受在线模仿学习和强化学习原理的启发,我们提出了多轮数据收集和训练策略,以赋予 LLM 在后续迭代中递归检测和纠正先前错误的能力。我们的实验表明,RISE 使 Llama2、Llama3 和 Mistral 模型在数学推理任务上能够通过更多轮次自我改进,在同等推理时间计算量下优于多种单轮策略。我们还发现 RISE 扩展性良好,常在更强模型上获得更大收益。分析表明,RISE 对回复做出有意义的改进,从而为具有挑战性的提示找到正确解,同时不会因表达更复杂分布而破坏单轮能力。

英文摘要:

A central piece in enabling intelligent agentic behavior in foundation models is to make them capable of introspecting upon their behavior, reasoning, and correcting their mistakes as more computation or interaction is available. Even the strongest proprietary large language models (LLMs) do not quite exhibit the ability of continually improving their responses sequentially, even in scenarios where they are explicitly told that they are making a mistake. In this paper, we develop RISE: Recursive IntroSpEction, an approach for fine-tuning LLMs to introduce this capability, despite prior work hypothesizing that this capability may not be possible to attain. Our approach prescribes an iterative fine-tuning procedure, which attempts to teach the model how to alter its response after having executed previously unsuccessful attempts to solve a hard test-time problem, with optionally additional environment feedback. RISE poses fine-tuning for a single-turn prompt as solving a multi-turn Markov decision process (MDP), where the initial state is the prompt. Inspired by principles in online imitation learning and reinforcement learning, we propose strategies for multi-turn data collection and training so as to imbue an LLM with the capability to recursively detect and correct its previous mistakes in subsequent iterations. Our experiments show that RISE enables Llama2, Llama3, and Mistral models to improve themselves with more turns on math reasoning tasks, outperforming several single-turn strategies given an equal amount of inference-time computation. We also find that RISE scales well, often attaining larger benefits with more capable models. Our analysis shows that RISE makes meaningful improvements to responses to arrive at the correct solution for challenging prompts, without disrupting one-turn abilities as a result of expressing more complex distributions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑