arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.10993cs.LGcs.AIcs.MA
收藏

基于线性函数逼近的多智能体拥塞代价最小化

Multi-Agent Congestion Cost Minimization With Linear Function Approximations

Indian Institute of Technology Bombay(孟买印度理工学院)

Prashant Trivedi, Nandyala Hemachandra

摘要:

这项工作认为多个代理穿越网络从源节点到目标节点。代理在链路上旅行的成本有私有和拥塞两个组成部分。智能体的目标是以分散的方式找到一条到达目标节点的总成本最小的路径。我们将其建模为一个完全分散的多智能体强化学习问题,并提出了一种新的多智能体拥塞成本最小化(MACCM)算法。我们的MACCM算法使用线性函数近似的转移概率和全局成本函数。在没有中央控制器的情况下,为了保护隐私,代理通过时变通信网络将成本函数参数传递给它们的邻居。此外,每个代理保持其估计的全局状态动作值,这是通过多代理扩展值迭代(MAEVI)子例程更新。我们表明,我们的MACCM算法实现了次线性遗憾。证明需要收敛的成本函数参数,MAEVI算法,并分析的遗憾界引起的MAEVI触发条件为每个代理。为了验证算法的有效性,我们在一个两节点多链路网络上实现了该算法,首先确定了最优策略,即每个周期内到达目标节点的最优代理数量。我们观察到,对于2个和3个代理人,平均遗憾接近于零。最优策略捕获停留在节点的最小成本和前往目标节点的拥塞成本之间的权衡。我们的工作是学习随机最短路问题的推广。

英文摘要:

This work considers multiple agents traversing a network from a source node to the goal node. The cost to an agent for traveling a link has a private as well as a congestion component. The agent's objective is to find a path to the goal node with minimum overall cost in a decentralized way. We model this as a fully decentralized multi-agent reinforcement learning problem and propose a novel multi-agent congestion cost minimization (MACCM) algorithm. Our MACCM algorithm uses linear function approximations of transition probabilities and the global cost function. In the absence of a central controller and to preserve privacy, agents communicate the cost function parameters to their neighbors via a time-varying communication network. Moreover, each agent maintains its estimate of the global state-action value, which is updated via a multi-agent extended value iteration (MAEVI) sub-routine. We show that our MACCM algorithm achieves a sub-linear regret. The proof requires the convergence of cost function parameters, the MAEVI algorithm, and analysis of the regret bounds induced by the MAEVI triggering condition for each agent. We implement our algorithm on a two node network with multiple links to validate it. We first identify the optimal policy, the optimal number of agents going to the goal node in each period. We observe that the average regret is close to zero for 2 and 3 agents. The optimal policy captures the trade-off between the minimum cost of staying at a node and the congestion cost of going to the goal node. Our work is a generalization of learning the stochastic shortest path problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑