Off-policy reinforcement learning enables near-optimal policy from suboptimal experience, thereby provisions opportunity for artificial intelligence applications in healthcare. Previous works have mainly framed patient-clinician interactions as Markov decision processes, while true physiological states are not necessarily fully observable from clinical data. We capture this situation with partially observable Markov decision process, in which an agent optimises its actions in a belief represented as a distribution of patient states inferred from individual history trajectories. A Gaussian mixture model is fitted for the observed data. Moreover, we take into account the fact that nuance in pharmaceutical dosage could presumably result in significantly different effect by modelling a continuous policy through a Gaussian approximator directly in the policy space, i.e. the actor. To address the challenge of infinite number of possible belief states which renders exact value iteration intractable, we evaluate and plan for only every encountered belief, through heuristic search tree by tightly maintaining lower and upper bounds of the true value of belief. We further resort to function approximations to update value bounds estimation, i.e. the critic, so that the tree search can be improved through more compact bounds at the fringe nodes that will be back-propagated to the root. Both actor and critic parameters are learned via gradient-based approaches. Our proposed policy trained from real intensive care unit data is capable of dictating dosing on vasopressors and intravenous fluids for sepsis patients that lead to the best patient outcomes.
POMDPPlanners: Open-Source Package for POMDP Planning
Faculty of Mathematics Technion - Israel Institute of Technology（数学系技术学院 - 以色列理工学院）；Stephen B. Klein Faculty of Aerospace Engineering Faculty of Data and Decision Sciences Technion - Israel Institute of Technology（Stephen B. Klein航空航天工程系数据与决策科学系技术学院 - 以色列理工学院）
Belief compression improves the tractability of large-scale partially observable Markov decision processes (POMDPs) by finding projections from high-dimensional belief space onto low-dimensional approximations, where solving to obtain action selection policies requires fewer computations. This paper develops a unified theoretical framework to analyse three existing linear belief compression approaches, including value-directed compression and two non-negative matrix factorisation (NMF) based algorithms. The results indicate that all the three known belief compression methods have their own cri…
Rather than augmenting rewards with penalties for undesired behavior, Constrained Partially Observable Markov Decision Processes (CPOMDPs) plan safely by imposing inviolable hard constraint value budgets. Previous work performing online planning for CPOMDPs has only been applied to discrete action and observation spaces. In this work, we propose algorithms for online CPOMDP planning for continuous state, action, and observation spaces by combining dual ascent with progressive widening. We empirically compare the effectiveness of our proposed algorithms on continuous CPOMDPs that model both toy…
IST Austria（奥地利科学技术研究所）；Université Libre de Bruxelles（布鲁塞尔自由大学）；University of Cambridge（剑桥大学）
A standard objective in partially-observable Markov decision processes (POMDPs) is to find a policy that maximizes the expected discounted-sum payoff. However, such policies may still permit unlikely but highly undesirable outcomes, which is problematic especially in safety-critical applications. Recently, there has been a surge of interest in POMDPs where the goal is to maximize the probability to ensure that the payoff is at least a given threshold, but these approaches do not consider any optimization beyond satisfying this threshold constraint. In this work we go beyond both the "expectati…
The partially observable Markov decision process (POMDP) is a principled general framework for robot decision making under uncertainty, but POMDP planning suffers from high computational complexity, when long-term planning is required. While temporally-extended macro-actions help to cut down the effective planning horizon and significantly improve computational efficiency, how do we acquire good macro-actions? This paper proposes Macro-Action Generator-Critic (MAGIC), which performs offline learning of macro-actions optimized for online POMDP planning. Specifically, MAGIC learns a macro-action…
Explanation through Reward Model Reconciliation using POMDP Tree Search
Department of Aerospace Engineering Sciences, University of Colorado Boulder（科罗拉多大学博尔德分校航空航天工程科学系）；Applied Physics Laboratory, Johns Hopkins University（约翰霍普金斯大学应用物理实验室）
Online POMDP Planning with Anytime Deterministic Optimality Guarantees
Technion Autonomous Systems Program (TASP)（技术学院自主系统计划）；Stephen B. Klein Faculty of Aerospace Engineering（斯蒂芬·B·克莱因航空航天工程学院）；Faculty of Data and Decision Sciences（数据与决策科学学院）
ETH Zürich（苏黎世联邦理工学院）；Swiss Federal Railways SBB（瑞士联邦铁路）；Technical University of Munich（慕尼黑工业大学）；Pennsylvania State Univ.（宾夕法尼亚州立大学）
Structural Health Monitoring (SHM) describes a process for inferring quantifiable metrics of structural condition, which can serve as input to support decisions on the operation and maintenance of infrastructure assets. Given the long lifespan of critical structures, this problem can be cast as a sequential decision making problem over prescribed horizons. Partially Observable Markov Decision Processes (POMDPs) offer a formal framework to solve the underlying optimal planning task. However, two issues can undermine the POMDP solutions. Firstly, the need for a model that can adequately describe…
Synthesizing POMDP Policies: Sampling Meets Model-checking via Learning
Nanyang Technological University, Singapore（新加坡南洋理工大学）；Tata Institute of Fundamental Research, Mumbai, India（印度孟买印度理工学院基础研究所以）；Université Libre de Bruxelles, Brussels, Belgium（比利时布鲁塞尔自由大学）；IITB Trust Lab, Department of CSE, IIT Bombay, Mumbai, India（印度孟买印度理工学院 Bombay 电子与计算机科学系信托实验室）
Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）
Actual causality and a closely related concept of responsibility attribution are central to accountable decision making. Actual causality focuses on specific outcomes and aims to identify decisions (actions) that were critical in realizing an outcome of interest. Responsibility attribution is complementary and aims to identify the extent to which decision makers (agents) are responsible for this outcome. In this paper, we study these concepts under a widely used framework for multi-agent sequential decision making under uncertainty: decentralized partially observable Markov decision processes …
HyPlan: Hybrid Learning-Assisted Planning Under Uncertainty for Safe Autonomous Driving
Saarland University, Computer Science Dept.（萨尔兰大学计算机科学系）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；French National Centre for Scientific Research (CNRS)（法国国家科学研究中心）
University of Cambridge（剑桥大学）；Toshiba Europe Limited（东芝欧洲有限公司）
Solving partially-observable Markov decision processes (POMDPs) is critical when applying reinforcement learning to real-world problems, where agents have an incomplete view of the world. We present graph convolutional memory (GCM), the first hybrid memory model for solving POMDPs using reinforcement learning. GCM uses either human-defined or data-driven topological priors to form graph neighborhoods, combining them into a larger network topology using dynamic programming. We query the graph using graph convolution, coalescing relevant memories into a context-dependent belief. When used withou…
California Institute of Technology（加州理工学院）；NASA Jet Propulsion Laboratory（NASA喷气推进实验室）
A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. Traditionally, policy synthesis techniques are proposed such that a total expected cost or reward is minimized or maximized. However, optimality in the total expected cost sense is only reasonable if system behavior in the large number of runs is of interest, which has limited the use of such policies in practical mission-critical scenarios, wherein large deviat…
Stanford University（斯坦福大学）；Cornell University（康奈尔大学）；Brain Of Things, Inc.（Brain Of Things公司）
The sense of touch, being the earliest sensory system to develop in a human body [1], plays a critical part of our daily interaction with the environment. In order to successfully complete a task, many manipulation interactions require incorporating haptic feedback. However, manually designing a feedback mechanism can be extremely challenging. In this work, we consider manipulation tasks that need to incorporate tactile sensor feedback in order to modify a provided nominal plan. To incorporate partial observation, we present a new framework that models the task as a partially observable Markov…
Health-related data is noisy and stochastic in implying the true physiological states of patients, limiting information contained in single-moment observations for sequential clinical decision making. We model patient-clinician interactions as partially observable Markov decision processes (POMDPs) and optimize sequential treatment based on belief states inferred from history sequence. To facilitate inference, we build a variational generative model and boost state representation with a recurrent neural network (RNN), incorporating an auxiliary loss from sequence auto-encoding. Meanwhile, we o…
Toward an Integrated Decision Making Framework for Optimized Stroke Diagnosis with DSA and Treatment under Uncertainty
Institut Teknologi Sepuluh Nopember (ITS)（十十一月科技学院（ITS））；Universitas Islam Negeri (UIN) Syarif Hidayatullah（赛义德·希达亚图拉伊斯兰国立大学）；Stanford University（斯坦福大学）
McGill University（麦吉尔大学）；Mila；Université de Montréal（蒙特利尔大学）；DeepMind
In this paper, we investigate learning temporal abstractions in cooperative multi-agent systems, using the options framework (Sutton et al, 1999). First, we address the planning problem for the decentralized POMDP represented by the multi-agent system, by introducing a \emph{common information approach}. We use the notion of \emph{common beliefs} and broadcasting to solve an equivalent centralized POMDP problem. Then, we propose the Distributed Option Critic (DOC) algorithm, which uses centralized option evaluation and decentralized intra-option improvement. We theoretically analyze the asympt…
Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observability
University of Virginia（弗吉尼亚大学）；Google（谷歌）；University of Maryland, College Park（马里兰大学 College Park 分校）；Stanford University（斯坦福大学）；Louisiana State University（路易斯安那州立大学）；College of William and Mary（威廉与玛丽学院）
Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management
organization= Department of Civil；Environmental Engineering , addressline= The Pennsylvania State University , city= University Park , postcode= 16802 , state= PA , country= USA；organization= Faculty of Architecture \& the Built Environment , addressline= Delft University of Technology , postcode= 2628 BL , state= Delft , country= The Netherlands
The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
University of Oxford（牛津大学）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Brown University（布朗大学）；University College London（伦敦大学学院）；University of Science and Technology of China（中国科学技术大学）；Imperial College London（伦敦帝国学院）；Dalian University of Technology（大连理工大学）；Chinese Academy of Sciences（中国科学院）；The Chinese University of Hong Kong（香港中文大学）；University of Georgia（佐治亚大学）；University of California, San Diego（加州大学圣地亚哥分校）；University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；University of Bristol（布里斯托大学）
