SimuDICE：通过世界模型更新与DICE估计的离线策略优化
SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation
AI总结：
针对离线强化学习中分布不匹配与样本有限的问题，提出SimuDICE框架，结合世界模型与DICE估计优化模拟经验采样，以更少的样本和规划步骤实现可比性能且鲁棒性强。
摘要：
在离线强化学习中，由于目标策略与用于收集数据的行为策略之间存在分布不匹配，且样本量有限，从预先收集的经验集中推导有效策略颇具挑战。基于模型的强化学习通过使用学习到的环境动力学模型生成模拟经验，提升了样本效率。然而，这些合成经验往往同样存在分布不匹配问题。为应对这些挑战，我们提出SimuDICE框架，该框架利用世界模型生成的合成经验，迭代优化从离线数据中得到的初始策略。SimuDICE基于平稳分布校正估计（DIstribution Correction Estimation, DICE）以及模型预测的估计置信度，调整状态-动作对的采样概率，从而提升这些模拟经验的质量。该方法通过平衡与常见经验相似的样本和存在分布不匹配的样本，引导策略改进。我们的实验表明，SimuDICE的性能与现有算法相当，但所需的预先收集经验和规划步骤更少，且在不同的数据收集策略下均保持鲁棒性。
英文摘要：
In offline reinforcement learning, deriving an effective policy from a pre-collected set of experiences is challenging due to the distribution mismatch between the target policy and the behavioral policy used to collect the data, as well as the limited sample size. Model-based reinforcement learning improves sample efficiency by generating simulated experiences using a learned dynamic model of the environment. However, these synthetic experiences often suffer from the same distribution mismatch. To address these challenges, we introduce SimuDICE, a framework that iteratively refines the initial policy derived from offline data using synthetically generated experiences from the world model. SimuDICE enhances the quality of these simulated experiences by adjusting the sampling probabilities of state-action pairs based on stationary DIstribution Correction Estimation (DICE) and the estimated confidence in the model's predictions. This approach guides policy improvement by balancing experiences similar to those frequently encountered with ones that have a distribution mismatch. Our experiments show that SimuDICE achieves performance comparable to existing algorithms while requiring fewer pre-collected experiences and planning steps, and it remains robust across varying data collection policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
