活在当下：学习适应不断演化策略的动力学模型
Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy
AI总结：
针对基于模型的强化学习中策略演化导致状态-动作分布偏移的问题，提出Policy-adapted Dynamics Model Learning (PDML)方法，通过动态调整历史策略混合分布使动力学模型持续适应演化策略，显著提升了样本效率和渐近性能。
摘要：
基于模型的强化学习（RL）通过学习动力学模型来为策略学习生成样本，在实践中通常比无模型RL具有更高的样本效率。以往的工作学习一个拟合所有历史策略的经验状态-动作访问分布（即样本回放缓冲区）的动力学模型。然而在本文中，我们观察到拟合所有历史策略分布的动力学模型，并不一定有利于对当前策略的模型预测，因为使用中的策略随时间不断演化。训练期间不断演化的策略会导致状态-动作访问分布偏移。我们从理论上分析了历史策略上的这种分布偏移如何影响模型学习和模型推演。随后，我们提出了一种新颖的动力学模型学习方法，名为Policy-adapted Dynamics Model Learning (PDML)。PDML动态调整历史策略混合分布，以确保学习到的模型能够持续适应不断演化策略的状态-动作访问分布。在MuJoCo的一系列连续控制环境中的实验表明，PDML与最先进的基于模型的RL方法相结合，在样本效率上取得了显著提升，并获得了更高的渐近性能。
英文摘要：
Model-based reinforcement learning (RL) often achieves higher sample efficiency in practice than model-free RL by learning a dynamics model to generate samples for policy learning. Previous works learn a dynamics model that fits under the empirical state-action visitation distribution for all historical policies, i.e., the sample replay buffer. However, in this paper, we observe that fitting the dynamics model under the distribution for \emph{all historical policies} does not necessarily benefit model prediction for the \emph{current policy} since the policy in use is constantly evolving over time. The evolving policy during training will cause state-action visitation distribution shifts. We theoretically analyze how this distribution shift over historical policies affects the model learning and model rollouts. We then propose a novel dynamics model learning method, named \textit{Policy-adapted Dynamics Model Learning (PDML)}. PDML dynamically adjusts the historical policy mixture distribution to ensure the learned model can continually adapt to the state-action visitation distribution of the evolving policy. Experiments on a range of continuous control environments in MuJoCo show that PDML achieves significant improvement in sample efficiency and higher asymptotic performance combined with the state-of-the-art model-based RL methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
