arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.08685cs.AI
收藏

RAMP:用于在线学习数字动作模型的混合深度强化学习

RAMP: Hybrid DRL for Online Learning of Numeric Action Models

Ben-Gurion University of the Negev(内盖夫本-古里安大学)

Yarin Benyamin, Argaman Mordoch, Shahaf S. Shperberg, Roni Stern

AI总结:

RAMP通过环境交互在线学习数字规划动作模型,结合深度强化学习策略、动作模型学习和规划,形成正反馈循环提升求解能力和计划质量。

摘要:

自动化规划算法需要指定每个动作的先决条件和效果的动作模型,但获取此类模型往往困难。从观察中学习动作模型是可行的,但现有算法在数值领域是离线的,需要专家轨迹作为输入。我们提出RAMP策略,通过与环境交互在线学习数值规划动作模型。RAMP同时训练深度强化学习(DRL)策略,从过去交互中学习数值动作模型,并在可能时使用该模型进行未来动作规划。这些组件形成正反馈循环:RL策略收集数据以细化动作模型,而规划器生成计划以继续训练RL策略。为了促进RL和数值规划的整合,我们开发了Numeric PDDLGym,一个将数值规划问题转换为Gym环境的自动化框架。在标准IPC数值领域实验结果表明,RAMP在求解能力和计划质量上显著优于PPO,一种广为人知的DRL算法。

英文摘要:

Automated planning algorithms require an action model specifying the preconditions and effects of each action, but obtaining such a model is often hard. Learning action models from observations is feasible, but existing algorithms for numeric domains are offline, requiring expert traces as input. We propose the Reinforcement learning, Action Model learning, and Planning (RAMP) strategy for learning numeric planning action models online via interactions with the environment. RAMP simultaneously trains a Deep Reinforcement Learning (DRL) policy, learns a numeric action model from past interactions, and uses that model to plan future actions when possible. These components form a positive feedback loop: the RL policy gathers data to refine the action model, while the planner generates plans to continue training the RL policy. To facilitate this integration of RL and numeric planning, we developed Numeric PDDLGym, an automated framework for converting numeric planning problems to Gym environments. Experimental results on standard IPC numeric domains show that RAMP significantly outperforms PPO, a well-known DRL algorithm, in terms of solvability and plan quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑