EnvGen：通过大语言模型生成并适配环境以训练具身智能体
EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents
AI总结：
该研究提出EnvGen框架，利用LLM生成并动态适配训练环境来强化小型RL智能体的薄弱技能，在Crafter、Heist环境中性能超GPT-4等SOTA方法，且LLM调用成本极低。
摘要：
当前通过交互实现具身学习的SOTA方法直接将大语言模型（LLMs）用作智能体，以决定环境中的下一步行动。凭借其世界知识与推理能力，LLM智能体比此前基于强化学习（RL）的小型智能体表现更优；但频繁调用LLM速度慢且成本高。我们并未直接将LLM用作智能体，而是提出疑问：能否利用LLM的推理能力自适应创建训练环境，帮助小型RL智能体学习其薄弱的有用技能？针对这一问题，我们提出了新型框架EnvGen。我们首先向LLM提供任务描述与智能体需要学习的模拟器目标，提示LLM生成训练环境，再要求其生成一组环境配置（例如不同地形、初始提供给智能体的物品等）。接下来，我们在原始环境与LLM生成环境的混合场景中训练小型RL智能体。随后，我们通过向LLM提供智能体的性能表现作为反馈，让LLM持续调整生成的环境，逐步提升智能体的薄弱技能。我们在Crafter和Heist环境中开展了全面实验，验证了EnvGen的有效性。研究发现，经EnvGen训练的小型RL智能体性能优于包括GPT-4智能体在内的SOTA方法，且学习长时序任务的速度显著更快。我们还证明，使用LLM动态调整环境的效果优于课程学习方法，并展示了环境如何随时间调整以帮助提升RL智能体的薄弱技能。此外，EnvGen的效率大幅提升：它仅需少量LLM调用（例如总共4次），而LLM智能体需要数千次调用。最后，我们针对EnvGen的设计选择开展了详细的消融实验。
英文摘要：
Recent SOTA approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their world knowledge and reasoning capabilities, LLM agents achieve stronger performance than previous smaller agents based on reinforcement learning (RL); however, frequently calling LLMs is slow and expensive. Instead of directly employing LLMs as agents, can we use LLMs' reasoning capabilities to adaptively create training environments to help smaller RL agents learn useful skills that they are weak at? We propose EnvGen, a novel framework to address this question. We first prompt an LLM to generate training environments by giving it the task description and simulator objectives that the agents should learn and then asking it to generate a set of environment configurations (e.g., different terrains, items initially given to agents, etc.). Next, we train a small RL agent in a mixture of the original and LLM-generated environments. Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent's performance. We demonstrate the usefulness of EnvGen with comprehensive experiments in Crafter and Heist environments. We find that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster. We also show that using an LLM to adapt environments dynamically outperforms curriculum learning approaches and how the environments are adapted to help improve RL agents' weaker skills over time. Additionally, EnvGen is substantially more efficient as it only uses a small number of LLM calls (e.g., 4 in total), whereas LLM agents require thousands of calls. Lastly, we present detailed ablation studies for EnvGen design choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
