CROP: 模型驱动离线策略优化的保守奖励
CROP: Conservative Reward for Model-based Offline Policy Optimization
AI总结：
本文提出CROP算法，通过同时最小化估计误差和随机动作奖励，实现保守奖励估计，缓解离线强化学习中的分布偏移问题。
摘要：
离线强化学习旨在利用收集的数据优化策略，而模型驱动方法因其能通过数据生成缓解数据覆盖限制而备受关注。然而，分布偏移导致的过估计是离线RL的主要挑战。本文提出保守奖励模型驱动离线策略优化算法（CROP），引入了简洁的目标，同时最小化估计误差和随机动作奖励，从而得到稳健的保守奖励估计器。理论分析表明，设计的保守奖励机制导致保守的策略评估并缓解分布偏移。实验显示，通过简单的奖励估计修改，CROP能够保守地估计奖励并实现与现有方法相竞争的性能。源代码将在接受后提供。
英文摘要：
Offline reinforcement learning (RL) aims to optimize a policy using collected data without online interactions. Model-based approaches are particularly appealing for addressing offline RL challenges because of their capability to mitigate the limitations of data coverage through data generation using models. Nonetheless, a prevalent issue in offline RL is the overestimation caused by distribution shift. This study proposes a novel model-based offline RL algorithm named Conservative Reward for model-based Offline Policy optimization (CROP). CROP introduces a streamlined objective that concurrently minimizes estimation error and the rewards of random actions, thereby yielding a robustly conservative reward estimator. Theoretical analysis shows that the designed conservative reward mechanism leads to a conservative policy evaluation and mitigates distribution shift. Experiments showcase that with the simple modification to reward estimation, CROP can conservatively estimate the reward and achieve competitive performance with existing methods. The source code will be available after acceptance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
