面向离线强化学习的目标条件数据增强方法
Goal-Conditioned Data Augmentation for Offline Reinforcement Learning
AI总结：
针对离线强化学习中次优数据集最优示范不足的问题，提出基于扩散模型的目标条件数据增强方法GODA，结合面向回报的目标条件、可控缩放及自适应门控机制生成高回报样本，在D4RL基准和交通信号控制任务上表现优于现有方法。
摘要：
离线强化学习（RL）可基于预先收集的离线数据集学习策略，无需直接与环境交互。但受离线数据集质量限制，在次优数据集中通常难以学习到合格的策略。针对最优示范不足的数据集问题，本文提出目标条件数据增强方法（Goal-cOnditioned Data Augmentation，GODA），这是一种基于扩散模型的新型目标条件方法，用于生成更高质量的增强样本。借助生成式建模的最新进展，GODA 引入了一种新型的面向回报的目标条件，并结合多种选择机制。具体而言，本文提出一种可控缩放技术，可在数据采样过程中提供增强的基于回报的引导。GODA 既能学习原始离线数据集的完整分布表征，又能生成带有选择性高回报目标的新数据，从而最大化有限最优示范的效用。此外，本文还提出一种新型自适应门控条件方法，用于处理含噪声的输入和条件，增强对目标导向引导的捕捉能力。本文在 D4RL 基准以及真实世界挑战（具体为交通信号控制（TSC）任务）上开展了实验，结果表明，在各类离线 RL 算法中，GODA 在提升数据质量方面效果显著，且性能优于当前最先进的数据增强方法。
英文摘要：
Offline reinforcement learning (RL) enables policy learning from pre-collected offline datasets, relaxing the need to interact directly with the environment. However, limited by the quality of offline datasets, it generally fails to learn well-qualified policies in suboptimal datasets. To address datasets with insufficient optimal demonstrations, we introduce Goal-cOnditioned Data Augmentation (GODA), a novel goal-conditioned diffusion-based method for augmenting samples with higher quality. Leveraging recent advancements in generative modelling, GODA incorporates a novel return-oriented goal condition with various selection mechanisms. Specifically, we introduce a controllable scaling technique to provide enhanced return-based guidance during data sampling. GODA learns a comprehensive distribution representation of the original offline datasets while generating new data with selectively higher-return goals, thereby maximizing the utility of limited optimal demonstrations. Furthermore, we propose a novel adaptive gated conditioning method for processing noisy inputs and conditions, enhancing the capture of goal-oriented guidance. We conduct experiments on the D4RL benchmark and real-world challenges, specifically traffic signal control (TSC) tasks, to demonstrate GODA's effectiveness in enhancing data quality and superior performance compared to state-of-the-art data augmentation methods across various offline RL algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
