通过奖励-策略协同进化实现高效的语言指导技能习得
Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution
AI总结：
提出奖励-策略协同进化框架，利用LLM动态生成奖励函数并结合改进的贝叶斯优化混合策略种群，以更少数据在高维机器人控制任务中实现95.3%的归一化改进。
摘要：
以最少的人类指导自主探索和解决任务的能力，对于具身智能的自我发展至关重要。尽管强化学习方法能在很大程度上减轻人类的工作量，但由于关节与任务之间的复杂关系，为现实世界任务设计奖励函数颇具挑战性，尤其是对于高维机器人控制。大型语言模型（LLMs）的最新进展使自动奖励函数设计成为可能。然而，通过从头重新训练策略来评估奖励函数的方法给奖励函数带来了过重的负担，期望其能在整个策略改进过程中始终有效。我们主张在机器人自主性中采用一种更实用的策略，侧重于使用依赖于策略的奖励函数来改进现有策略，而非使用通用奖励函数。为此，我们提出了一种新颖的奖励-策略协同进化框架，其中奖励函数和已学习的策略从彼此实时的渐进改进中获益，从而实现更高效、更高性能的技能习得。具体而言，奖励进化过程将机器人先前的最佳奖励函数、任务描述和环境转换为文本输入。这些输入用于查询 LLMs 以生成动态数量的奖励函数候选，确保在每轮进化中持续改进。在策略进化方面，我们的方法通过将历史最优策略和随机策略进行混合来生成新的策略种群。通过改进的贝叶斯优化，我们的方法高效且鲁棒地识别出能力最强且可塑性最高的奖励-策略组合，随后进入下一轮协同进化。尽管使用了较少的数据，我们的方法在各种高维机器人技能学习任务中展现了 95.3% 的平均归一化改进。
英文摘要：
The ability to autonomously explore and resolve tasks with minimal human guidance is crucial for the self-development of embodied intelligence. Although reinforcement learning methods can largely ease human effort, it's challenging to design reward functions for real-world tasks, especially for high-dimensional robotic control, due to complex relationships among joints and tasks. Recent advancements large language models (LLMs) enable automatic reward function design. However, approaches evaluate reward functions by re-training policies from scratch placing an undue burden on the reward function, expecting it to be effective throughout the whole policy improvement process. We argue for a more practical strategy in robotic autonomy, focusing on refining existing policies with policy-dependent reward functions rather than a universal one. To this end, we propose a novel reward-policy co-evolution framework where the reward function and the learned policy benefit from each other's progressive on-the-fly improvements, resulting in more efficient and higher-performing skill acquisition. Specifically, the reward evolution process translates the robot's previous best reward function, descriptions of tasks and environment into text inputs. These inputs are used to query LLMs to generate a dynamic amount of reward function candidates, ensuring continuous improvement at each round of evolution. For policy evolution, our method generates new policy populations by hybridizing historically optimal and random policies. Through an improved Bayesian optimization, our approach efficiently and robustly identifies the most capable and plastic reward-policy combination, which then proceeds to the next round of co-evolution. Despite using less data, our approach demonstrates an average normalized improvement of 95.3% across various high-dimensional robotic skill learning tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
