慎思行动、线上规划与学习的层级运作模式
Deliberative Acting, Online Planning and Learning with Hierarchical Operational Models
摘要：
在人工智能研究中，合成行动计划通常使用抽象地指定作为行动结果可能发生的行为的描述性模型，并为有效计算状态转换而量身定制。然而，执行计划的行动需要操作模型，其中丰富的计算控制结构和闭环在线决策用于指定如何在不确定的执行上下文中执行动作，对事件做出反应并适应展开的情况。深思熟虑的行动者，整合了行动和计划，通常需要同时使用这两种模型--这在试图开发不同的模型、验证它们的一致性以及平滑地交错行动和计划时会导致问题。作为替代方案，我们定义并实现了一个集成的代理和规划系统，其中规划和代理使用相同的操作模型。这些依赖于分层的面向任务的细化方法，提供丰富的控制结构。动作组件，被称为反应式动作引擎（RAE），灵感来自著名的PRS系统。在每个决策步骤，RAE可以从规划者那里得到关于效用函数的接近最优选择的建议。随时计划器使用类似于UCT的蒙特卡罗树搜索过程，称为UPOM，其展开是对参与者的操作模型的模拟。我们还提出了使用RAE和UPOM的学习策略，从在线表演经验和/或模拟规划结果中获得从决策上下文到方法实例的映射以及引导UPOM的启发式函数。我们证明了UPOM的渐近收敛到静态域中的最优方法，并通过实验表明，UPOM和学习策略显着提高代理效率和鲁棒性。
英文摘要：
In AI research, synthesizing a plan of action has typically used descriptive models of the actions that abstractly specify what might happen as a result of an action, and are tailored for efficiently computing state transitions. However, executing the planned actions has needed operational models, in which rich computational control structures and closed-loop online decision-making are used to specify how to perform an action in a nondeterministic execution context, react to events and adapt to an unfolding situation. Deliberative actors, which integrate acting and planning, have typically needed to use both of these models together -- which causes problems when attempting to develop the different models, verify their consistency, and smoothly interleave acting and planning. As an alternative, we define and implement an integrated acting and planning system in which both planning and acting use the same operational models. These rely on hierarchical task-oriented refinement methods offering rich control structures. The acting component, called Reactive Acting Engine (RAE), is inspired by the well-known PRS system. At each decision step, RAE can get advice from a planner for a near-optimal choice with respect to a utility function. The anytime planner uses a UCT-like Monte Carlo Tree Search procedure, called UPOM, whose rollouts are simulations of the actor's operational models. We also present learning strategies for use with RAE and UPOM that acquire, from online acting experiences and/or simulated planning results, a mapping from decision contexts to method instances as well as a heuristic function to guide UPOM. We demonstrate the asymptotic convergence of UPOM towards optimal methods in static domains, and show experimentally that UPOM and the learning strategies significantly improve the acting efficiency and robustness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
