不确定性驱动的轨迹截断用于离线强化学习中的数据增强
Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning
AI总结：
针对离线强化学习中模型生成样本不可靠的问题，提出不确定性驱动的轨迹截断方法TATU，通过累积不确定性截断合成轨迹，理论证明性能界限，并在D4RL基准上显著提升多种离线RL算法的性能。
摘要：
配备训练好的环境动力学模型后，基于模型的离线强化学习（RL）算法通常能够从固定大小的数据集中成功学习到好的策略，即使某些数据集质量较差。然而，不幸的是，无法保证从训练好的动力学模型生成的样本是可靠的（例如，一些合成样本可能位于静态数据集的支持区域之外）。为了解决这个问题，我们提出了带有不确定性的轨迹截断（TATU），当沿轨迹累积的不确定性过大时，自适应地截断合成轨迹。我们从理论上展示了TATU的性能界限，以证明其优势。为了实证展示TATU的优势，我们首先将其与两种经典的基于模型的离线RL算法MOPO和COMBO结合。此外，我们将TATU与几种现成的无模型离线RL算法（如BCQ）集成。在D4RL基准上的实验结果表明，TATU显著提升了它们的性能，通常提升幅度很大。代码可在此处获取。
英文摘要：
Equipped with the trained environmental dynamics, model-based offline reinforcement learning (RL) algorithms can often successfully learn good policies from fixed-sized datasets, even some datasets with poor quality. Unfortunately, however, it can not be guaranteed that the generated samples from the trained dynamics model are reliable (e.g., some synthetic samples may lie outside of the support region of the static dataset). To address this issue, we propose Trajectory Truncation with Uncertainty (TATU), which adaptively truncates the synthetic trajectory if the accumulated uncertainty along the trajectory is too large. We theoretically show the performance bound of TATU to justify its benefits. To empirically show the advantages of TATU, we first combine it with two classical model-based offline RL algorithms, MOPO and COMBO. Furthermore, we integrate TATU with several off-the-shelf model-free offline RL algorithms, e.g., BCQ. Experimental results on the D4RL benchmark show that TATU significantly improves their performance, often by a large margin. Code is available here.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
