基于代理状态的保守策略迭代:在具有保证收敛性的Dec-POMDPs中求解风险寻求策略
Risk-seeking conservative policy iteration with agent-state based policies for Dec-POMDPs with guaranteed convergence
AI总结:
本文提出一种迭代最佳响应算法,通过修改目标函数在Dec-POMDP中实现风险寻求与保守策略迭代,以在有限内存下获得局部最优解。
摘要:
最优解决分布式决策问题,建模为Dec-POMDPs,已知是NEXP完全问题。这些最优解是基于代理完整历史观察和动作的策略。然而,某些应用由于计算能力限制,可能需要更紧凑的策略,这可以通过考虑有限数量的记忆状态(或代理状态)来建模。虽然基于代理状态的策略类可能不包含最优解,但仍有必要在该类中寻找最佳代理状态策略。我们专注于一种迭代最佳响应风格的算法,该算法保证单调改进并在多项式时间内收敛到局部最优。为了获得更好的局部最优,我们使用修改的目标函数,激励风险寻求与保守策略迭代更新。我们的实证结果表明,我们的方法在多个基准Dec-POMDPs上表现与最先进的方法相当,尽管内存有限,仍能实现近最优性能。我们还显示,使用更多的代理状态(更大的内存)会导致更大的性能提升。我们的方法为在Dec-POMDP问题中将内存约束纳入代理提供了一种新方法。
英文摘要:
Optimally solving decentralized decision-making problems modeled as Dec-POMDPs is known to be NEXP-complete. These optimal solutions are policies based on the entire history of observations and actions of an agent. However, some applications may require more compact policies because of limited compute capabilities, which can be modeled by considering a limited number of memory states (or agent states). While such an agent-state based policy class may not contain the optimal solution, it is still of practical interest to find the best agent-state policy within the class. We focus on an iterated best response style algorithm which guarantees monotonic improvements and convergence to a local optimum in polynomial runtime in the Dec-POMDP model size. In order to obtain a better local optimum, we use a modified objective which incentivizes risk-seeking alongside a conservative policy iteration update. Our empirical results show that our approach performs as well as state-of-the-art approaches on several benchmark Dec-POMDPs, achieving near-optimal performance while having polynomial runtime despite the limited memory. We also show that using more agent states (a larger memory) leads to greater performance. Our approach provides a novel way of incorporating memory constraints on the agents in the Dec-POMDP problem.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。