不要在安全性上妥协：面向约束离线强化学习的扩散正则化方法
Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RL
AI总结：
针对离线约束强化学习的安全与性能平衡问题，提出DRCORL方法，通过扩散模型提取策略并结合梯度操纵实现安全适配，在机器人任务中兼具可靠安全性、快速推理与优异奖励表现，实用性强。
摘要：
约束强化学习（RL）旨在在安全约束下寻求高性能策略。我们聚焦于智能体仅拥有固定数据集的离线设置——这在现实任务中很常见，用于避免不安全的探索。为解决这一问题，我们提出扩散正则化约束离线强化学习（DRCORL），该方法首先使用扩散模型从离线数据中捕捉行为策略，随后提取简化策略以实现高效推理。我们进一步应用梯度操纵进行安全适配，平衡奖励目标与约束满足度。该方法利用高质量离线数据的同时纳入了安全要求。实验结果表明，DRCORL在机器人学习任务中实现了可靠的安全性能、快速推理和优异的奖励表现。与现有安全离线RL方法相比，它始终满足成本限制，且在相同超参数下表现良好，展现出在现实场景中的实用适用性。
英文摘要：
Constrained reinforcement learning (RL) seeks high-performance policies under safety constraints. We focus on an offline setting where the agent has only a fixed dataset -- common in realistic tasks to prevent unsafe exploration. To address this, we propose Diffusion-Regularized Constrained Offline Reinforcement Learning (DRCORL), which first uses a diffusion model to capture the behavioral policy from offline data and then extracts a simplified policy to enable efficient inference. We further apply gradient manipulation for safety adaptation, balancing the reward objective and constraint satisfaction. This approach leverages high-quality offline data while incorporating safety requirements. Empirical results show that DRCORL achieves reliable safety performance, fast inference, and strong reward outcomes across robot learning tasks. Compared to existing safe offline RL methods, it consistently meets cost limits and performs well with the same hyperparameters, indicating practical applicability in real-world scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
