arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.04363cs.LGcs.AIstat.ML
收藏

通过多步前驱模型缓解Dyna规划中的价值幻觉

Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models

Dept of Computer Science, Harvey Mudd College, USA(美国哈维穆德学院计算机科学系)

Farzane Aminmansour, Taher Jafferjee, Ehsan Imani, Erin Talvitie, Micheal Bowling, Martha White

AI总结:

本文提出通过多步前驱模型改进Dyna算法,以减少环境模型误差导致的价值幻觉问题,实验验证了该方法的有效性。

摘要:

Dyna风格的强化学习(RL)代理通过使用环境模型生成的模拟经验来更新价值函数,从而提高样本效率。然而,学习准确的环境动力学模型往往困难,即使是小误差也可能导致Dyna代理失败。本文指出,这种失败的一个潜在原因是基于模拟状态的价值进行预测,因此提出了一种新的Dyna算法来避免此问题。讨论了基于后继或前驱模型(正向或反向模拟)以及一步或多步更新的设计空间,探讨了三种变体,但第四种变体(使用前驱模型进行多步更新)却未被探索。提出了“幻觉价值假设”(HVH):将真实状态的价值更新到模拟状态的价值可能导致误导的动作价值,从而影响控制策略。评估了四种Dyna变体,其中三种将真实状态更新到模拟状态(可能指向幻觉值),而本文提出的方法则不进行此操作。实验结果支持了HVH,并表明使用前驱模型进行多步更新是开发更鲁棒于模型误差的Dyna算法的有希望方向。

英文摘要:

Dyna-style reinforcement learning (RL) agents improve sample efficiency over model-free RL agents by updating the value function with simulated experience generated by an environment model. However, it is often difficult to learn accurate models of environment dynamics, and even small errors may result in failure of Dyna agents. In this paper, we highlight that one potential cause of that failure is bootstrapping off of the values of simulated states, and introduce a new Dyna algorithm to avoid this failure. We discuss a design space of Dyna algorithms, based on using successor or predecessor models -- simulating forwards or backwards -- and using one-step or multi-step updates. Three of the variants have been explored, but surprisingly the fourth variant has not: using predecessor models with multi-step updates. We present the \emph{Hallucinated Value Hypothesis} (HVH): updating the values of real states towards values of simulated states can result in misleading action values which adversely affect the control policy. We discuss and evaluate all four variants of Dyna amongst which three update real states toward simulated states -- so potentially toward hallucinated values -- and our proposed approach, which does not. The experimental results provide evidence for the HVH, and suggest that using predecessor models with multi-step updates is a promising direction toward developing Dyna algorithms that are more robust to model error.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑