针对对抗性状态扰动的信念增强悲观Q学习
Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations
AI总结：
针对强化学习智能体易受对抗状态扰动的问题，提出融合信念状态推断与扩散状态纯化的悲观Q学习算法，在强攻击下表现优异且训练开销与正则化方法相当。
摘要：
强化学习（RL）已在诸多领域取得显著成功。然而，其数据驱动的特性也带来了可被恶意对手利用的新漏洞。近期研究表明，训练良好的RL智能体在测试阶段很容易因状态观测遭到策略性扰动而被操控。现有解决方案要么引入正则项以提升训练后策略抗扰动的平滑性，要么采用对抗训练同时训练智能体策略与攻击者策略。但前者无法为强攻击提供足够防护，后者在大型环境中计算成本过高。本文提出一种新的鲁棒RL算法，通过推导悲观策略来防范智能体对真实状态的不确定性。该方法进一步结合信念状态推断与基于扩散的状态纯化技术以降低不确定性。实验结果表明，我们的方法在强攻击下表现优异，且训练开销与基于正则化的方法相当。代码可在https://github.com/SliencerX/Belief-enriched-robust-Q-learning获取。
英文摘要：
Reinforcement learning (RL) has achieved phenomenal success in various domains. However, its data-driven nature also introduces new vulnerabilities that can be exploited by malicious opponents. Recent work shows that a well-trained RL agent can be easily manipulated by strategically perturbing its state observations at the test stage. Existing solutions either introduce a regularization term to improve the smoothness of the trained policy against perturbations or alternatively train the agent's policy and the attacker's policy. However, the former does not provide sufficient protection against strong attacks, while the latter is computationally prohibitive for large environments. In this work, we propose a new robust RL algorithm for deriving a pessimistic policy to safeguard against an agent's uncertainty about true states. This approach is further enhanced with belief state inference and diffusion-based state purification to reduce uncertainty. Empirical results show that our approach obtains superb performance under strong attacks and has a comparable training overhead with regularization-based methods. Our code is available at https://github.com/SliencerX/Belief-enriched-robust-Q-learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
