ASTRO: 通过动力学引导的轨迹滚出实现自适应缝合
ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rollouts
AI总结：
ASTRO通过动态引导的轨迹滚出生成新颖且一致的轨迹，提升离线强化学习中策略学习的性能。
摘要：
离线强化学习（RL）使智能体能够从预收集的数据集中学习最优策略。然而，包含次优和碎片化轨迹的数据集对奖励传播提出了挑战，导致价值估计不准确和策略性能下降。虽然通过生成模型进行轨迹缝合提供了一个有前途的解决方案，但现有增强方法经常产生轨迹，这些轨迹要么局限于行为策略的支持域，要么违反了底层动力学，从而限制了它们在策略改进中的有效性。我们提出了ASTRO，一个数据增强框架，用于生成分布上新颖且动力学一致的轨迹用于离线RL。ASTRO首先学习了一个时间-距离表示来识别不同的且可达的缝合目标。然后，我们采用一个动力学引导的缝合规划器，通过滚出偏差反馈（定义为执行预测动作时目标状态序列与实际到达状态序列之间的差距）来适应性地生成连接动作序列，以提高轨迹缝合的可行性和可达性。这种方法通过缝合实现了有效的增强，并最终提高了策略学习。ASTRO在各种算法上均优于先前的离线RL增强方法，在具有挑战性的OGBench套件上取得了显著的性能提升，并在标准离线RL基准如D4RL上展示了持续的改进。
英文摘要：
Offline reinforcement learning (RL) enables agents to learn optimal policies from pre-collected datasets. However, datasets containing suboptimal and fragmented trajectories present challenges for reward propagation, resulting in inaccurate value estimation and degraded policy performance. While trajectory stitching via generative models offers a promising solution, existing augmentation methods frequently produce trajectories that are either confined to the support of the behavior policy or violate the underlying dynamics, thereby limiting their effectiveness for policy improvement. We propose ASTRO, a data augmentation framework that generates distributionally novel and dynamics-consistent trajectories for offline RL. ASTRO first learns a temporal-distance representation to identify distinct and reachable stitch targets. We then employ a dynamics-guided stitch planner that adaptively generates connecting action sequences via Rollout Deviation Feedback, defined as the gap between target state sequence and the actual arrived state sequence by executing predicted actions, to improve trajectory stitching's feasibility and reachability. This approach facilitates effective augmentation through stitching and ultimately enhances policy learning. ASTRO outperforms prior offline RL augmentation methods across various algorithms, achieving notable performance gain on the challenging OGBench suite and demonstrating consistent improvements on standard offline RL benchmarks such as D4RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
