arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.02005cs.AIcs.MAcs.SYeess.SY
收藏

风险感知的分布式多智能体强化学习

Risk-Aware Distributed Multi-Agent Reinforcement Learning

University of Washington(华盛顿大学);Western Washington University(西华盛顿大学);Washington University in St. Louis(圣路易斯华盛顿大学)

Abdullah Al Maruf, Luyao Niu, Bhaskar Ramasubramanian, Andrew Clark, Radha Poovendran

AI总结:

本文提出一种基于条件风险价值(CVaR)的分布式多智能体强化学习方法(CVaR QD-Learning算法),通过学习风险感知动作解决未知环境下的决策问题,证明了Bellman算子的收敛性与智能体价值函数的共识性,并通过仿真验证了风险参数的影响。

摘要:

自主网络和赛博物理系统需要在未知环境中进行决策、学习和控制。此类决策对多种因素敏感,包括建模误差、成本变化以及概率分布尾部事件的影响。尽管多智能体强化学习(MARL)提供了一个通过最小化平均成本来学习行为的框架,但它不足以克服上述挑战。在本文中,我们开发了一种分布式MARL方法,通过学习风险感知动作来解决未知环境中的决策问题。我们使用条件风险价值(CVaR)来表征被最小化的成本函数,并定义了一个Bellman算子来表征与给定状态-动作对相关的价值函数。我们证明了该算子满足收缩性质,并且它收敛于最优价值函数。然后,我们提出了一种名为CVaR QD-Learning算法的分布式MARL算法,并证明了各个智能体的价值函数能够达成共识。我们识别了在实现CVaR QD-Learning算法时出现的几个挑战,并提出了解决方案以克服这些挑战。我们通过仿真评估了CVaR QD-Learning算法,并展示了风险参数对达成共识时的价值函数的影响。

英文摘要:

Autonomous cyber and cyber-physical systems need to perform decision-making, learning, and control in unknown environments. Such decision-making can be sensitive to multiple factors, including modeling errors, changes in costs, and impacts of events in the tails of probability distributions. Although multi-agent reinforcement learning (MARL) provides a framework for learning behaviors through repeated interactions with the environment by minimizing an average cost, it will not be adequate to overcome the above challenges. In this paper, we develop a distributed MARL approach to solve decision-making problems in unknown environments by learning risk-aware actions. We use the conditional value-at-risk (CVaR) to characterize the cost function that is being minimized, and define a Bellman operator to characterize the value function associated to a given state-action pair. We prove that this operator satisfies a contraction property, and that it converges to the optimal value function. We then propose a distributed MARL algorithm called the CVaR QD-Learning algorithm, and establish that value functions of individual agents reaches consensus. We identify several challenges that arise in the implementation of the CVaR QD-Learning algorithm, and present solutions to overcome these. We evaluate the CVaR QD-Learning algorithm through simulations, and demonstrate the effect of a risk parameter on value functions at consensus.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑