基于语言模型预测控制从人类反馈中学习更快地学习
Learning to Learn Faster from Human Feedback with Language Model Predictive Control
AI总结:
本文提出语言模型预测控制(LMPC)框架,通过微调PaLM 2训练转移动力学模型,结合MPC发现成功捷径,显著提高机器人对人类反馈的适应效率及在未见任务和实体上的成功率。
摘要:
大型语言模型(LLMs)已被证明具备广泛的能力,例如根据语言指令编写机器人代码——这使得非专家能够指挥机器人行为、根据反馈修改行为,或将它们组合以执行新任务。然而,这些由上下文学习驱动的能力仅限于短期交互,用户的反馈只有在符合 LLM 上下文大小的情况下才保持相关性,并且在更长的交互中可能会被遗忘。在本文中,我们研究了微调编写机器人代码的 LLMs,使其记住上下文交互并提高其可教导性,即它们适应人类输入的效率(通过用户认为任务成功前的平均纠正次数来衡量)。我们的关键观察是,当人机交互被视为一个部分可观测马尔可夫决策过程(其中人类语言输入是观测,机器人代码输出是动作)时,训练 LLM 完成先前的交互就是训练一个转移动力学模型——该模型可以与经典机器人技术(如模型预测控制(MPC))相结合,以发现通往成功的更短路径。这催生了语言模型预测控制(LMPC),这是一个微调 PaLM 2 的框架,旨在提高其在 5 种机器人实体上的 78 项任务的可教导性——将未见任务的非专家教学成功率提高了 26.9%,同时将平均人类纠正次数从 2.4 次减少到 1.9 次。实验表明,LMPC 产生了强大的元学习器,将基于上下文学习在未见机器人实体和 API 上的新任务成功率提高了 31.5%。视频、代码和演示请见:https://robot-teaching.github.io/。
英文摘要:
Large language models (LLMs) have been shown to exhibit a wide range of capabilities, such as writing robot code from language commands -- enabling non-experts to direct robot behaviors, modify them based on feedback, or compose them to perform new tasks. However, these capabilities (driven by in-context learning) are limited to short-term interactions, where users' feedback remains relevant for only as long as it fits within the context size of the LLM, and can be forgotten over longer interactions. In this work, we investigate fine-tuning the robot code-writing LLMs, to remember their in-context interactions and improve their teachability i.e., how efficiently they adapt to human inputs (measured by average number of corrections before the user considers the task successful). Our key observation is that when human-robot interactions are viewed as a partially observable Markov decision process (in which human language inputs are observations, and robot code outputs are actions), then training an LLM to complete previous interactions is training a transition dynamics model -- that can be combined with classic robotics techniques such as model predictive control (MPC) to discover shorter paths to success. This gives rise to Language Model Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its teachability on 78 tasks across 5 robot embodiments -- improving non-expert teaching success rates of unseen tasks by 26.9% while reducing the average number of human corrections from 2.4 to 1.9. Experiments show that LMPC also produces strong meta-learners, improving the success rate of in-context learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos, code, and demos at: https://robot-teaching.github.io/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。