arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.02652cs.LGcs.AIcs.CR
收藏

训练时攻击的自适应折扣

Adaptive Discounting of Training Time Attacks

Nanyang Technological University(南洋理工大学)

Ridhima Bector, Abhay Aradhya, Chai Quek, Zinovi Rabinovich

AI总结:

本研究提出一种名为gammaDDPG的DDPG算法变体,通过动态调整攻击策略规划视野,实现对受害者无法自然采取的目标行为的建设性训练时攻击,并验证了其有效性。

摘要:

对强化学习(RL)解决方案最隐蔽的攻击之一是训练时攻击(TTAs),它会在学习到的行为中制造漏洞和后门。建设性TTAs(C-TTAs)现已出现,不仅限于简单的破坏,攻击者还能在训练RL智能体(受害者)时强制其执行特定的目标行为。然而,即使是最先进的C-TTAs,也仅关注那些若非环境动力学的特定特征被C-TTAs利用,受害者本可自然采取的目标行为。在本研究中,我们证明,即使目标行为因环境动力学及相对于受害者目标的非最优性而无法被采取,C-TTA也是可能的。为了在此背景下寻找有效的攻击,我们开发了DDPG算法的一种特殊变体,称之为gammaDDPG,用于学习这种更强版本的C-TTA。gammaDDPG根据受害者当前行为动态改变攻击策略的规划视野。这改善了整个攻击时间线上的努力分配,并降低了攻击者对受害者不确定性的影响。为展示我们方法的特性并更好地将结果与先前研究联系起来,我们在实验中采用了最先进C-TTA的3D网格域。代码可在“bit.ly/github-rb-gDDPG”获取。

英文摘要:

Among the most insidious attacks on Reinforcement Learning (RL) solutions are training-time attacks (TTAs) that create loopholes and backdoors in the learned behaviour. Not limited to a simple disruption, constructive TTAs (C-TTAs) are now available, where the attacker forces a specific, target behaviour upon a training RL agent (victim). However, even state-of-the-art C-TTAs focus on target behaviours that could be naturally adopted by the victim if not for a particular feature of the environment dynamics, which C-TTAs exploit. In this work, we show that a C-TTA is possible even when the target behaviour is un-adoptable due to both environment dynamics as well as non-optimality with respect to the victim objective(s). To find efficient attacks in this context, we develop a specialised flavour of the DDPG algorithm, which we term gammaDDPG, that learns this stronger version of C-TTA. gammaDDPG dynamically alters the attack policy planning horizon based on the victim's current behaviour. This improves effort distribution throughout the attack timeline and reduces the effect of uncertainty the attacker has about the victim. To demonstrate the features of our method and better relate the results to prior research, we borrow a 3D grid domain from a state-of-the-art C-TTA for our experiments. Code is available at "bit.ly/github-rb-gDDPG".

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑