供应链中的隐私保护多智能体强化学习
Privacy Preserving Multi-Agent Reinforcement Learning in Supply Chains
AI总结:
针对供应链MARL中的策略数据隐私问题,本文基于EzPC上的SecFloat实现安全MPC,并重构MADDPG前后向运算,在保护隐私的同时显著降低浪费并提升收益。
摘要:
本文解决多智能体强化学习(MARL)中的隐私问题,特别是在供应链场景中,个体的战略性数据必须保密。供应链中的组织被建模为智能体,每个智能体在与其他智能体交互的同时寻求优化自身目标。由于每个组织的策略取决于相邻组织的策略,因此保护状态和动作相关信息的隐私至关重要。为应对这一挑战,我们提出一种博弈论的隐私保护机制,在MARL环境中使用安全多方计算(MPC)框架。我们的主要贡献是成功实现了基于EzPC的安全MPC框架SecFloat,以解决该问题。然而,仅使用SecFloat实现MADDPG等策略梯度方法的操作,虽然在概念上可行,但在程序实现上却难以处理。为克服这一障碍,我们设计了一种新方法,将神经网络的前向传播和反向传播分解为与SecFloat兼容的基本操作,从而创建高效且安全的MADDPG算法版本。此外,我们提出了一种以隐私保护方式执行浮点运算的学习机制,这是MARL框架中成功学习的重要特征。实验表明,与不共享数据相比,在2 PC中供应链浪费平均减少68.19%,同时每个参与者的平均累计收益平均提高42.27%。本工作为实用的隐私保护MARL铺平了道路,有望显著改进供应链场景乃至更广泛领域中的安全计算。
英文摘要:
This paper addresses privacy concerns in multi-agent reinforcement learning (MARL), specifically within the context of supply chains where individual strategic data must remain confidential. Organizations within the supply chain are modeled as agents, each seeking to optimize their own objectives while interacting with others. As each organization's strategy is contingent on neighboring strategies, maintaining privacy of state and action-related information is crucial. To tackle this challenge, we propose a game-theoretic, privacy-preserving mechanism, utilizing a secure multi-party computation (MPC) framework in MARL settings. Our major contribution is the successful implementation of a secure MPC framework, SecFloat on EzPC, to solve this problem. However, simply implementing policy gradient methods such as MADDPG operations using SecFloat, while conceptually feasible, would be programmatically intractable. To overcome this hurdle, we devise a novel approach that breaks down the forward and backward pass of the neural network into elementary operations compatible with SecFloat , creating efficient and secure versions of the MADDPG algorithm. Furthermore, we present a learning mechanism that carries out floating point operations in a privacy-preserving manner, an important feature for successful learning in MARL framework. Experiments reveal that there is on average 68.19% less supply chain wastage in 2 PC compared to no data share, while also giving on average 42.27% better average cumulative revenue for each player. This work paves the way for practical, privacy-preserving MARL, promising significant improvements in secure computation within supply chain contexts and broadly.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。