We consider the stochastic shortest path planning problem in MDPs, i.e., the problem of designing policies that ensure reaching a goal state from a given initial state with minimum accrued cost. In order to account for rare but important realizations of the system, we consider a nested dynamic coherent risk total cost functional rather than the conventional risk-neutral total expected cost. Under some assumptions, we show that optimal, stationary, Markovian policies exist and can be found via a special Bellman's equation. We propose a computational technique based on difference convex programs (DCPs) to find the associated value functions and therefore the risk-averse policies. A rover navigation MDP is used to illustrate the proposed methodology with conditional-value-at-risk (CVaR) and entropic-value-at-risk (EVaR) coherent risk measures.
California Institute of Technology（加州理工学院）；NASA Jet Propulsion Laboratory（美国国家航空航天局喷气推进实验室）
We consider the problem of designing policies for partially observable Markov decision processes (POMDPs) with dynamic coherent risk objectives. Synthesizing risk-averse optimal policies for POMDPs requires infinite memory and thus undecidable. To overcome this difficulty, we propose a method based on bounded policy iteration for designing stochastic but finite state (memory) controllers, which takes advantage of standard convex optimization methods. Given a memory budget and optimality criterion, the proposed method modifies the stochastic finite state controller leading to sub-optimal soluti…
Sequential decision problems in applications such as manipulation in warehouses, multi-step meal preparation, and routing in autonomous vehicle networks often involve reasoning about uncertainty, planning over discrete modes as well as continuous states, and reacting to dynamic updates. To formalize such problems generally, we introduce a class of Markov Decision Processes (MDPs) called Dynamic Multimodal Stochastic Shortest Paths (DMSSPs). Much of the work in these domains solves deterministic variants, which can yield poor results when the uncertainty has downstream effects. We develop a Hyb…
Stochastic sequential decision making often requires hierarchical structure in the problem where each high-level action should be further planned with primitive states and actions. In addition, many real-world applications require a plan that satisfies constraints on the secondary costs such as risk measure or fuel consumption. In this paper, we propose a hierarchical constrained stochastic shortest path problem (HC-SSP) that meets those two crucial requirements in a single framework. Although HC-SSP provides a useful framework to model such planning requirements in many real-world application…
Oxford Robotics Institute, University of Oxford（牛津大学牛津机器人研究所）
Previous work on planning as active inference addresses finite horizon problems and solutions valid for online planning. We propose solving the general Stochastic Shortest-Path Markov Decision Process (SSP MDP) as probabilistic inference. Furthermore, we discuss online and offline methods for planning under uncertainty. In an SSP MDP, the horizon is indefinite and unknown a priori. SSP MDPs generalize finite and infinite horizon MDPs and are widely used in the artificial intelligence community. Additionally, we highlight some of the differences between solving an MDP using dynamic programming …
Planning in Markov decision processes (MDPs) typically optimises the expected cost. However, optimising the expectation does not consider the risk that for any given run of the MDP, the total cost received may be unacceptably high. An alternative approach is to find a policy which optimises a risk-averse objective such as conditional value at risk (CVaR). However, optimising the CVaR alone may result in poor performance in expectation. In this work, we begin by showing that there can be multiple policies which obtain the optimal CVaR. This motivates us to propose a lexicographic approach which…
College of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）
Route planning is important in transportation. Existing works focus on finding the shortest path solution or using metrics such as safety and energy consumption to determine the planning. It is noted that most of these studies rely on prior knowledge of road network, which may be not available in certain situations. In this paper, we design a route planning algorithm based on deep reinforcement learning (DRL) for pedestrians. We use travel time consumption as the metric, and plan the route by predicting pedestrian flow in the road network. We put an agent, which is an intelligent robot, on a v…
This paper presents sensitivity analyses of resilience-based active distribution system planning solutions with respect to different parameters. The distribution system planning problem is formulated as a two-stage risk-averse stochastic optimization model with conditional value-at-risk (CVaR) as the risk measure. The probabilistic scenarios are obtained using regional wind profiles, and Monte Carlo simulations are conducted to obtain failure scenarios based on component fragility models. The planning measure includes advanced distribution grid operations with intentional islanding measures. T…
This work presents a methodology to incorporate reliability constraints in the optimal power systems expansion planning problem. Besides LOLP and EPNS, traditionally used in power systems, this work proposes the use of the risk measures VaR (Value-at-Risk) and CVaR (Conditional Value-at-Risk), widely used in financial markets. The explicit consideration of reliability constraints in the planning problem can be an extremely hard task and, in order to minimize computational effort, this work applies the Benders decomposition technique splitting the expansion planning problem into an investment p…
We investigate the problem of risk averse robot path planning using the deep reinforcement learning and distributionally robust optimization perspectives. Our problem formulation involves modelling the robot as a stochastic linear dynamical system, assuming that a collection of process noise samples is available. We cast the risk averse motion planning problem as a Markov decision process and propose a continuous reward function design that explicitly takes into account the risk of collision with obstacles while encouraging the robot's motion towards the goal. We learn the risk-averse robot co…
Masaryk University（马萨里克大学）；Institute of Science and Technology Austria（奥地利科学技术研究所）
Markov decision processes (MDPs) are the defacto frame-work for sequential decision making in the presence ofstochastic uncertainty. A classical optimization criterion forMDPs is to maximize the expected discounted-sum pay-off, which ignores low probability catastrophic events withhighly negative impact on the system. On the other hand,risk-averse policies require the probability of undesirableevents to be below a given threshold, but they do not accountfor optimization of the expected payoff. We consider MDPswith discounted-sum payoff with failure states which repre-sent catastrophic outcomes…
Massachusetts Institute of Technology（麻省理工学院）；Toyota Research Institute（丰田研究院）
Risk-bounded motion planning is an important yet difficult problem for safety-critical tasks. While existing mathematical programming methods offer theoretical guarantees in the context of constrained Markov decision processes, they either lack scalability in solving larger problems or produce conservative plans. Recent advances in deep reinforcement learning improve scalability by learning policy networks as function approximators. In this paper, we propose an extension of soft actor critic model to estimate the execution risk of a plan through a risk critic and produce risk-bounded policies …
In this paper, we consider planning in stochastic shortest path (SSP) problems, a subclass of Markov Decision Problems (MDP). We focus on medium-size problems whose state space can be fully enumerated. This problem has numerous important applications, such as navigation and planning under uncertainty. We propose a new approach for constructing a multi-level hierarchy of progressively simpler abstractions of the original problem. Once computed, the hierarchy can be used to speed up planning by first finding a policy for the most abstract level and then recursively refining it into a solution to…
Autonomous agents such as self-driving cars or parcel robots need to recognize and avoid possible collisions with obstacles in order to move successfully in their environment. Humans, however, have learned to predict movements intuitively and to avoid obstacles in a forward-looking way. The task of collision avoidance can be divided into a global and a local level. Regarding the global level, we propose an approach called "Predictive Collision Management Path Planning" (PCMP). At the local level, solutions for collision avoidance are used that prevent an inevitable collision. Therefore, the ai…
Oxford Robotics Institute, University of Oxford（牛津大学牛津机器人研究所）
The parameters for a Markov Decision Process (MDP) often cannot be specified exactly. Uncertain MDPs (UMDPs) capture this model ambiguity by defining sets which the parameters belong to. Minimax regret has been proposed as an objective for planning in UMDPs to find robust policies which are not overly conservative. In this work, we focus on planning for Stochastic Shortest Path (SSP) UMDPs with uncertain cost and transition functions. We introduce a Bellman equation to compute the regret for a policy. We propose a dynamic programming algorithm that utilises the regret Bellman equation, and sho…
Real-world multi-agent planning problems cannot be solved using decision-theoretic planning methods due to the exponential complexity. We approximate firefighting in rescue simulation as a spatially distributed task and model with multi-agent Markov decision process. We use recent approximation methods for spatial task problems to reduce the model complexity. Our approximations are single-agent, static task, shortest path pruning, dynamic planning horizon, and task clustering. We create scenarios from RoboCup Rescue Simulation maps and evaluate our methods on these graph worlds. The results sh…
Intel Labs（英特尔实验室）；Technion Israel Institute of Technology（以色列理工学院）
Offline reinforcement-learning (RL) algorithms learn to make decisions using a given, fixed training dataset without online data collection. This problem setting is captivating because it holds the promise of utilizing previously collected datasets without any costly or risky interaction with the environment. However, this promise also bears the drawback of this setting as the restricted dataset induces uncertainty because the agent can encounter unfamiliar sequences of states and actions that the training data did not cover. To mitigate the destructive uncertainty effects, we need to balance …
A risk-averse preview-based $Q$-learning planner is presented for navigation of autonomous vehicles. To this end, the multi-lane road ahead of a vehicle is represented by a finite-state non-stationary Markov decision process (MDP). A risk assessment unit module is then presented that leverages the preview information provided by sensors along with a stochastic reachability module to assign reward values to the MDP states and update them as scenarios develop. A sampling-based risk-averse preview-based Q-learning algorithm is finally developed that generates samples using the preview information…
Norwegian University of Science and Technology (NTNU)（挪威科技大学）
We consider warm-started optimized trajectory planning for autonomous surface vehicles (ASVs) by combining the advantages of two types of planners: an A* implementation that quickly finds the shortest piecewise linear path, and an optimal control-based trajectory planner. A nonlinear 3-degree-of-freedom underactuated model of an ASV is considered, along with an objective functional that promotes energy-efficient and readily observable maneuvers. The A* algorithm is guaranteed to find the shortest piecewise linear path to the goal position based on a uniformly decomposed map. Dynamic informatio…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；UCLA Vision Lab（加州大学洛杉矶分校视觉实验室）；Robotics and Mechanisms Laboratory（机器人与机械实验室）；Laboratory for Embedded Machines and Ubiquitous Robots（嵌入式机器与普适机器人实验室）
In this paper, we propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties for safer navigation through cluttered environments. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates through each step of our MPC's finite time horizon. The RNN model is trained on a dataset that comprises of robot and landmark poses generated from camera images and inertial measurement unit (IMU) readings via a state-of-the-art visual-inertial…
In remote or islanded communities, the use of microgrids (MGs) is necessary to ensure electrification and resilience of supply. However, even in small-scale systems, it is computationally and mathematically challenging to design low-cost, optimal, sustainable solutions taking into consideration all the uncertainties of load demands and power generations from renewable energy sources (RESs). This paper uses the open-source Python-based Energy Planning (PyEPLAN) tool, developed for the design of sustainable MGs in remote areas, on the Alderney island, the 3$^{rd}$ largest of the Channel Islands …
To eliminate the effect of ocean currents when addressing the optimal path in the underwater environment, an intelligent algorithm designed for the unmanned underwater vehicle (UUV) is proposed in this paper. The algorithm consists of two parts: a neural network-based algorithm that deducts the shortest path and avoids all possible collisions; and an adjusting component that balances off the deviation brought by the effect of ocean currents. The optimization results of the proposed algorithm are presented in detail, and compared with the path planning algorithm that does not consider the effec…
We present improvements to a recently developed method for trajectory planning for autonomous surface vehicles (ASVs) in terms of run time. The original method combines two types of planners: An A* implementation that quickly finds the global shortest piecewise linear path on a uniformly discretized map, and an optimal control-based trajectory planner which takes into account ASV dynamics. Firstly, we propose an improvement to the discretization of the map by switching to a Voronoi diagram rather than the uniform discretization, which offers a far more sparse search tree for the A* implementat…
