迈向鲁棒策略：通过对抗攻击与防御增强离线强化学习
Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
AI总结：
针对离线强化学习策略鲁棒性不足、易受观测扰动和攻击影响的问题，提出一种结合对抗攻击与防御的框架，通过训练时扰动观测并以防御为正则项增强策略，在D4RL基准上验证了防御的有效性。
摘要：
离线强化学习（RL）通过在海量离线数据上预训练策略，解决了强化学习中数据探索成本高、风险高的固有难题，可直接在真实环境中部署或微调。然而，这种训练范式可能会削弱策略的鲁棒性，导致在实际场景中因观测扰动或蓄意攻击而出现性能下降。尽管对抗攻击与防御在深度学习领域已得到广泛研究，但它们在离线强化学习中的应用仍较为有限。本文提出了一个利用先进对抗攻击与防御技术提升离线强化学习模型鲁棒性的框架。该框架在训练期间通过扰动观测来攻击智能体（actor）和评价器（critic）组件，并将对抗防御作为正则化项来增强所学策略的性能。论文引入了四种攻击方法和两种防御方法，并在D4RL基准上进行了评估。结果表明，actor和critic均易受攻击，而防御方法在提升策略鲁棒性方面效果显著。该框架有望提升离线强化学习模型在实际场景中的可靠性。
英文摘要：
Offline reinforcement learning (RL) addresses the challenge of expensive and high-risk data exploration inherent in RL by pre-training policies on vast amounts of offline data, enabling direct deployment or fine-tuning in real-world environments. However, this training paradigm can compromise policy robustness, leading to degraded performance in practical conditions due to observation perturbations or intentional attacks. While adversarial attacks and defenses have been extensively studied in deep learning, their application in offline RL is limited. This paper proposes a framework to enhance the robustness of offline RL models by leveraging advanced adversarial attacks and defenses. The framework attacks the actor and critic components by perturbing observations during training and using adversarial defenses as regularization to enhance the learned policy. Four attacks and two defenses are introduced and evaluated on the D4RL benchmark. The results show the vulnerability of both the actor and critic to attacks and the effectiveness of the defenses in improving policy robustness. This framework holds promise for enhancing the reliability of offline RL models in practical scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
