arXiv 2503.202852025-03-26cs.LGcs.AI

基于模型的离线强化学习与对抗性数据增强

Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation

National Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）；City University of Hong Kong（香港城市大学）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机系）

Hongye Cao, Fan Feng, Jing Huo, Shangdong Yang, Meng Fang, Tianpei Yang, Yang Gao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对离线强化学习中静态数据导致策略不稳健的问题，提出MORAL方法，利用对抗性数据增强和集成模型交替采样扩展训练数据，并引入差分因子正则化，在D4RL基准上优于现有方法。

摘要：

基于模型的离线强化学习（RL）从离线数据集中构建环境模型，以执行保守的策略优化。现有方法侧重于通过集成模型学习状态转移，并展开保守估计以减轻外推误差。然而，静态数据使得开发稳健的策略具有挑战性，且离线智能体无法访问环境以收集新数据。为解决这些挑战，我们引入了基于模型的离线强化学习与对抗性数据增强（MORAL）。在MORAL中，我们通过采用对抗性数据增强来替代固定时域展开，利用集成模型执行交替采样以丰富训练数据。具体而言，这一对抗过程动态地选择针对策略的集成模型进行有偏采样，减轻固定模型的乐观估计，从而稳健地扩展用于策略优化的训练数据。此外，在对抗过程中集成一个差分因子进行正则化，确保外推中的误差最小化。这种数据增强优化适应各种离线任务，无需调整展开时域，展现出显著的应用性。在D4RL基准上的大量实验表明，MORAL在策略学习和样本效率方面优于其他基于模型的离线RL方法。

英文摘要：

Model-based offline Reinforcement Learning (RL) constructs environment models from offline datasets to perform conservative policy optimization. Existing approaches focus on learning state transitions through ensemble models, rollouting conservative estimation to mitigate extrapolation errors. However, the static data makes it challenging to develop a robust policy, and offline agents cannot access the environment to gather new data. To address these challenges, we introduce Model-based Offline Reinforcement learning with AdversariaL data augmentation (MORAL). In MORAL, we replace the fixed horizon rollout by employing adversaria data augmentation to execute alternating sampling with ensemble models to enrich training data. Specifically, this adversarial process dynamically selects ensemble models against policy for biased sampling, mitigating the optimistic estimation of fixed models, thus robustly expanding the training data for policy optimization. Moreover, a differential factor is integrated into the adversarial process for regularization, ensuring error minimization in extrapolations. This data-augmented optimization adapts to diverse offline tasks without rollout horizon tuning, showing remarkable applicability. Extensive experiments on D4RL benchmark demonstrate that MORAL outperforms other model-based offline RL methods in terms of policy learning and sample efficiency.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。