arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01309cs.NEcs.AI
收藏

REvolve:利用人类反馈的大语言模型奖励进化

REvolve: Reward Evolution with Large Language Models using Human Feedback

Centre for Applied Autonomous Sensor Systems (AASS)(应用自主传感器系统中心)

Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, Pedro Zuidberg Dos Martires

AI总结:

REvolve利用人类反馈引导大语言模型进化生成奖励函数,在自动驾驶、人形运动和灵巧操作中优于现有基线。

摘要:

设计有效的奖励函数对于训练强化学习(RL)算法至关重要。然而,由于某些任务具有主观性且难以显式量化,即使对于领域专家而言,这一设计也并非易事。在近期工作中,大语言模型(LLMs)已被用于从自然语言任务描述中生成奖励,利用其广泛的指令微调和对人类行为的常识性理解。在本工作中,我们假设在人类反馈引导下,LLMs可被用于制定反映人类隐性知识的奖励函数。我们在三个具有挑战性的场景中研究这一问题——自动驾驶、人形机器人运动和灵巧操作——在这些场景中,“良好”行为的概念是隐性的且难以量化。为此,我们引入了REvolve,一个真正进化的框架,利用LLMs进行RL中的奖励设计。REvolve通过利用人类反馈引导进化过程来生成和细化奖励函数,有效地将人类的隐性知识转化为用于训练(深度)RL智能体的显式奖励函数。在实验中,我们证明了使用REvolve设计的奖励训练的智能体优于其他最先进的基线方法。

英文摘要:

Designing effective reward functions is crucial to training reinforcement learning (RL) algorithms. However, this design is non-trivial, even for domain experts, due to the subjective nature of certain tasks that are hard to quantify explicitly. In recent works, large language models (LLMs) have been used for reward generation from natural language task descriptions, leveraging their extensive instruction tuning and commonsense understanding of human behavior. In this work, we hypothesize that LLMs, guided by human feedback, can be used to formulate reward functions that reflect human implicit knowledge. We study this in three challenging settings -- autonomous driving, humanoid locomotion, and dexterous manipulation -- wherein notions of ``good" behavior are tacit and hard to quantify. To this end, we introduce REvolve, a truly evolutionary framework that uses LLMs for reward design in RL. REvolve generates and refines reward functions by utilizing human feedback to guide the evolution process, effectively translating implicit human knowledge into explicit reward functions for training (deep) RL agents. Experimentally, we demonstrate that agents trained on REvolve-designed rewards outperform other state-of-the-art baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑