arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.06255cs.MAcs.AIcs.LG
收藏

隐私工程化价值分解网络用于合作多智能体强化学习

Privacy-Engineered Value Decomposition Networks for Cooperative Multi-Agent Reinforcement Learning

University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Florida(佛罗里达大学)

Parham Gohari, Matthew Hale, Ufuk Topcu

AI总结:

针对合作多智能体强化学习中的隐私问题,提出PE-VDN算法,通过分布式计算、多方计算和差分隐私技术,在保护环境数据的同时实现80%的Vanilla VDN胜率。

摘要:

在合作多智能体强化学习(Co-MARL)中,一个智能体团队必须共同优化团队的长期奖励以学习指定任务。作为团队优化奖励通常需要智能体间的通信和数据共享,这可能导致潜在的隐私问题。我们假设隐私考量禁止智能体共享其环境交互数据。因此,我们提出隐私工程化价值分解网络(PE-VDN),一种Co-MARL算法,它在建模多智能体协调的同时,可证明地保护智能体环境交互数据的机密性。我们整合了三种隐私工程技术来重新设计VDN算法的数据流,VDN是一种现有的Co-MARL算法,它整合智能体的环境交互数据以训练一个建模多智能体协调的中央控制器,并开发了PE-VDN。在第一种技术中,我们设计了一种分布式计算方案,消除了Vanilla VDN对共享环境交互数据的依赖。然后,我们利用隐私保护的多方计算协议来保证分布式计算方案的数据流不会带来新的隐私风险。最后,我们强制执行差分隐私,以预防针对智能体训练数据(即过去的环境交互)的推断威胁,当智能体基于其神经网络预测采取行动时。我们在星际争霸多智能体竞赛(SMAC)中实现了PE-VDN,并表明它在保持提供有意义隐私保证的差分隐私水平的同时,达到了Vanilla VDN胜率的80%。结果表明,PE-VDN能够在不牺牲多智能体协调的情况下保护智能体环境交互数据的机密性。

英文摘要:

In cooperative multi-agent reinforcement learning (Co-MARL), a team of agents must jointly optimize the team's long-term rewards to learn a designated task. Optimizing rewards as a team often requires inter-agent communication and data sharing, leading to potential privacy implications. We assume privacy considerations prohibit the agents from sharing their environment interaction data. Accordingly, we propose Privacy-Engineered Value Decomposition Networks (PE-VDN), a Co-MARL algorithm that models multi-agent coordination while provably safeguarding the confidentiality of the agents' environment interaction data. We integrate three privacy-engineering techniques to redesign the data flows of the VDN algorithm, an existing Co-MARL algorithm that consolidates the agents' environment interaction data to train a central controller that models multi-agent coordination, and develop PE-VDN. In the first technique, we design a distributed computation scheme that eliminates Vanilla VDN's dependency on sharing environment interaction data. Then, we utilize a privacy-preserving multi-party computation protocol to guarantee that the data flows of the distributed computation scheme do not pose new privacy risks. Finally, we enforce differential privacy to preempt inference threats against the agents' training data, past environment interactions, when they take actions based on their neural network predictions. We implement PE-VDN in StarCraft Multi-Agent Competition (SMAC) and show that it achieves 80% of Vanilla VDN's win rate while maintaining differential privacy levels that provide meaningful privacy guarantees. The results demonstrate that PE-VDN can safeguard the confidentiality of agents' environment interaction data without sacrificing multi-agent coordination.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑