基于分散式多智能体强化学习的当日送达问题
A Decentralised Multi-Agent Reinforcement Learning Approach for the Same-Day Delivery Problem
摘要:
近几年来,当日送达服务越来越受欢迎。这些通常已被建模为以前的研究作为一个特定的类的动态车辆路径问题(DVRP),货物必须从一个仓库交付给一组客户在同一天的订单。DVRP的自适应精确解方法即使对于小问题实例也可能变得棘手。在本文中,我们将SDDP制定为马尔可夫决策过程(MDP),并使用参数共享的深度Q网络来解决它,这对应于分散的多智能体强化学习(MARL)方法。为此,我们创建了一个多代理基于网格的SDD环境,包括多个车辆,一个中央仓库和动态订单生成。此外,我们还引入了特定区域的订单生成和奖励概率。我们比较我们提出的MARL方法对混合间编程(MIP)的解决方案的性能。结果表明,我们提出的MARL框架执行与基于MIP的政策时,订单数量相对较低。对于具有高阶到达率的问题实例,计算结果表明MARL方法的性能低于MIP高达30%。当采用区域特定参数时,两种方法之间的性能差距变得更小。对于具有30个订单的5x5网格场景,差距从30%减少到3%。执行时间结果表明,MARL方法平均比基于MIP的策略快65倍,因此对于实时控制可能更有利,至少对于小型实例而言。
英文摘要:
Same-Day Delivery services are becoming increasingly popular in recent years. These have been usually modelled by previous studies as a certain class of Dynamic Vehicle Routing Problem (DVRP) where goods must be delivered from a depot to a set of customers in the same day that the orders were placed. Adaptive exact solution methods for DVRPs can become intractable even for small problem instances. In this paper, we formulate the SDDP as a Markov Decision Process (MDP) and solve it using a parameter-sharing Deep Q-Network, which corresponds to a decentralised Multi-Agent Reinforcement Learning (MARL) approach. For this, we create a multi-agent grid-based SDD environment, consisting of multiple vehicles, a central depot and dynamic order generation. In addition, we introduce zone-specific order generation and reward probabilities. We compare the performance of our proposed MARL approach against a Mixed Inter Programming (MIP) solution. Results show that our proposed MARL framework performs on par with MIP-based policy when the number of orders is relatively low. For problem instances with higher order arrival rates, computational results show that the MARL approach underperforms the MIP by up to 30%. The performance gap between both methods becomes smaller when zone-specific parameters are employed. The gap is reduced from 30% to 3% for a 5x5 grid scenario with 30 orders. Execution time results indicate that the MARL approach is, on average, 65 times faster than the MIP-based policy, and therefore may be more advantageous for real-time control, at least for small-sized instances.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。