谁是最强的敌人?深度RL中的最优有效规避攻击
Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL
摘要:
评估强化学习(RL)代理在最强/最优对抗扰动下的最坏情况下的性能对于理解RL代理的鲁棒性至关重要。然而,找到最佳的对手是具有挑战性的,无论是在我们是否可以找到最佳的攻击和如何有效地找到它。现有的对抗性强化学习的工作要么使用基于策略的方法,可能无法找到最强的对手,或直接训练一个基于RL的对手,通过治疗代理作为环境的一部分,它可以找到最佳的对手,但可能会变得难以处理在一个大的状态空间。本文提出了一种新的攻击方法,通过设计一个名为“演员”的功能和一个名为“导演”的基于RL的学习器之间的合作,以找到最佳的攻击。参与者为给定的策略扰动方向制作状态扰动,而导演学习提出最佳策略扰动方向。我们提出的算法PA-AD在理论上是最优的,并且在具有大状态空间的环境中比之前基于RL的工作效率要高得多。实证结果表明,我们提出的PA-AD普遍优于国家的最先进的攻击方法在各种Atari和MuJoCo环境。通过将PA-AD应用于对抗训练,我们在强大对手的多个任务中实现了最先进的经验鲁棒性。代码库在https://github.com/umd-huang-lab/paad_adv_rl上发布。
英文摘要:
Evaluating the worst-case performance of a reinforcement learning (RL) agent under the strongest/optimal adversarial perturbations on state observations (within some constraints) is crucial for understanding the robustness of RL agents. However, finding the optimal adversary is challenging, in terms of both whether we can find the optimal attack and how efficiently we can find it. Existing works on adversarial RL either use heuristics-based methods that may not find the strongest adversary, or directly train an RL-based adversary by treating the agent as a part of the environment, which can find the optimal adversary but may become intractable in a large state space. This paper introduces a novel attacking method to find the optimal attacks through collaboration between a designed function named "actor" and an RL-based learner named "director". The actor crafts state perturbations for a given policy perturbation direction, and the director learns to propose the best policy perturbation directions. Our proposed algorithm, PA-AD, is theoretically optimal and significantly more efficient than prior RL-based works in environments with large state spaces. Empirical results show that our proposed PA-AD universally outperforms state-of-the-art attacking methods in various Atari and MuJoCo environments. By applying PA-AD to adversarial training, we achieve state-of-the-art empirical robustness in multiple tasks under strong adversaries. The codebase is released at https://github.com/umd-huang-lab/paad_adv_rl.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。