LLM驱动的策略扩散：提升离线强化学习的泛化能力
LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning
AI总结：
针对离线强化学习泛化难的问题，提出结合文本与轨迹提示的LLMDPD方法，用LLM和transformer分别编码两类提示作为策略扩散模型条件，在未见任务上超越了现有最优离线RL方法。
摘要：
强化学习（RL）以其强大的决策能力著称，已被广泛应用于各类现实场景。然而，随着离线数据集的日益丰富，加之人类专家难以设计完善的在线环境，离线RL中的泛化挑战愈发凸显。受限于离线数据的固有局限，仅基于收集到的经验训练的RL智能体往往难以泛化到新任务或新环境。为解决这一挑战，我们提出LLM-Driven Policy Diffusion（LLMDPD，大语言模型驱动的策略扩散）——一种利用任务特定提示提升离线RL泛化能力的新方法。我们的方法同时结合基于文本的任务描述和轨迹提示来引导策略学习：利用大语言模型（LLM）处理文本提示，借助其自然语言理解能力和海量知识库提供丰富的任务相关上下文；同时使用transformer模型编码轨迹提示，捕捉底层转移动力学中的结构化行为模式。这些提示作为上下文感知的策略级扩散模型的条件输入，使RL智能体能够有效泛化到未见过的任务。实验结果表明，LLMDPD在未见过的任务上优于当前最优的离线RL方法，凸显了其在多样场景中提升泛化性和适应性的有效性。
英文摘要：
Reinforcement Learning (RL) is known for its strong decision-making capabilities and has been widely applied in various real-world scenarios. However, with the increasing availability of offline datasets and the lack of well-designed online environments from human experts, the challenge of generalization in offline RL has become more prominent. Due to the limitations of offline data, RL agents trained solely on collected experiences often struggle to generalize to new tasks or environments. To address this challenge, we propose LLM-Driven Policy Diffusion (LLMDPD), a novel approach that enhances generalization in offline RL using task-specific prompts. Our method incorporates both text-based task descriptions and trajectory prompts to guide policy learning. We leverage a large language model (LLM) to process text-based prompts, utilizing its natural language understanding and extensive knowledge base to provide rich task-relevant context. Simultaneously, we encode trajectory prompts using a transformer model, capturing structured behavioral patterns within the underlying transition dynamics. These prompts serve as conditional inputs to a context-aware policy-level diffusion model, enabling the RL agent to generalize effectively to unseen tasks. Our experimental results demonstrate that LLMDPD outperforms state-of-the-art offline RL methods on unseen tasks, highlighting its effectiveness in improving generalization and adaptability in diverse settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
