部分可观测马尔可夫决策过程（POMDP）能够对随机且不确定环境下的复杂序贯决策问题进行建模。阻碍其在现实应用中被广泛采用的一个主要原因，是缺少合适的 POMDP 模型或其模拟器。可用的求解算法（如强化学习，RL）需要知道转移动态和观测生成过程，而这些往往是未知的，并且推断起来并不简单。在本工作中，我们提出了一个通过深度 RL 对 POMDP 进行推断和鲁棒求解的联合框架。首先，所有转移模型和观测模型参数通过对以动作为条件的隐马尔可夫模型进行 Markov Chain Monte Carlo 采样来联合推断，以便从可用数据中恢复完整的后验分布。随后，采用深度 RL 技术求解参数不确定的 POMDP，并通过域随机化将参数分布纳入求解过程，以开发对模型不确定性具有鲁棒性的解决方案。作为进一步贡献，我们将构成无模型 RL 解决方案的 transformers 和长短期记忆网络，与一种基于模型/无模型混合方法进行了比较。我们将这些方法应用于铁路资产最优维护规划这一现实问题。
英文摘要：
Partially Observable Markov Decision Processes (POMDPs) can model complex sequential decision-making problems under stochastic and uncertain environments. A main reason hindering their broad adoption in real-world applications is the lack of availability of a suitable POMDP model or a simulator thereof. Available solution algorithms, such as Reinforcement Learning (RL), require the knowledge of the transition dynamics and the observation generating process, which are often unknown and non-trivial to infer. In this work, we propose a combined framework for inference and robust solution of POMDPs via deep RL. First, all transition and observation model parameters are jointly inferred via Markov Chain Monte Carlo sampling of a hidden Markov model, which is conditioned on actions, in order to recover full posterior distributions from the available data. The POMDP with uncertain parameters is then solved via deep RL techniques with the parameter distributions incorporated into the solution via domain randomization, in order to develop solutions that are robust to model uncertainty. As a further contribution, we compare the use of transformers and long short-term memory networks, which constitute model-free RL solutions, with a model-based/model-free hybrid approach. We apply these methods to the real-world problem of optimal maintenance planning for railway assets.
ETH Zürich（苏黎世联邦理工学院）；Swiss Federal Railways SBB（瑞士联邦铁路）；Technical University of Munich（慕尼黑工业大学）；Pennsylvania State Univ.（宾夕法尼亚州立大学）
Structural Health Monitoring (SHM) describes a process for inferring quantifiable metrics of structural condition, which can serve as input to support decisions on the operation and maintenance of infrastructure assets. Given the long lifespan of critical structures, this problem can be cast as a sequential decision making problem over prescribed horizons. Partially Observable Markov Decision Processes (POMDPs) offer a formal framework to solve the underlying optimal planning task. However, two issues can undermine the POMDP solutions. Firstly, the need for a model that can adequately describe…
University of Liege（列日大学）；The Pennsylvania State University（宾夕法尼亚州立大学）；Aalborg University（奥尔堡大学）
Civil and maritime engineering systems, among others, from bridges to offshore platforms and wind turbines, must be efficiently managed as they are exposed to deterioration mechanisms throughout their operational life, such as fatigue or corrosion. Identifying optimal inspection and maintenance policies demands the solution of a complex sequential decision-making problem under uncertainty, with the main objective of efficiently controlling the risk associated with structural failures. Addressing this complexity, risk-based inspection planning methodologies, supported often by dynamic Bayesian …
Solving Truly Massive Budgeted Monotonic POMDPs with Oracle-Guided Meta-Reinforcement Learning
Department of Aerospace Engineering（航空航天工程系）；University of Illinois Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；Department of Mathematics（数学系）；Engineer Research and Development Center（工程研究与发展中心）；U.S. Army Corps of Engineers（美国陆军工程兵团）
Determination of inspection and maintenance policies for minimizing long-term risks and costs in deteriorating engineering environments constitutes a complex optimization problem. Major computational challenges include the (i) curse of dimensionality, due to exponential scaling of state/action set cardinalities with the number of components; (ii) curse of history, related to exponentially growing decision-trees with the number of decision-steps; (iii) presence of state uncertainties, induced by inherent environment stochasticity and variability of inspection/monitoring measurements; (iv) prese…
In reinforcement learning, agents have successfully used environments modeled with Markov decision processes (MDPs). However, in many problem domains, an agent may suffer from noisy observations or random times until its subsequent decision. While partially observable Markov decision processes (POMDPs) have dealt with noisy observations, they have yet to deal with the unknown time aspect. Of course, one could discretize the time, but this leads to Bellman's Curse of Dimensionality. To incorporate continuous sojourn-time distributions in the agent's decision making, we propose that partially ob…
Adaptive digital twins for predictive decision-making: Online Bayesian learning of transition dynamics
MOX – Dipartimento di Matematica, Politecnico di Milano（MOX——数学系，米兰理工大学）；Dipartimento di Ingegneria Civile e Ambientale, Politecnico di Milano（土木工程与环境工程系，米兰理工大学）；Mathematics Department, Emory University（埃默里大学数学系）
Decision makers involved in the management of civil assets and systems usually take actions under constraints imposed by societal regulations. Some of these constraints are related to epistemic quantities, as the probability of failure events and the corresponding risks. Sensors and inspectors can provide useful information supporting the control process (e.g. the maintenance process of an asset), and decisions about collecting this information should rely on an analysis of its cost and value. When societal regulations encode an economic perspective that is not aligned with that of the decisio…
University of Technology Sydney（悉尼科技大学）；JD AI Research（京东人工智能研究院）；Sydney Trains-Operational Technology（悉尼列车运营技术部）
Railway points are among the key components of railway infrastructure. As a part of signal equipment, points control the routes of trains at railway junctions, having a significant impact on the reliability, capacity, and punctuality of rail transport. Traditionally, maintenance of points is based on a fixed time interval or raised after the equipment failures. Instead, it would be of great value if we could forecast points' failures and take action beforehand, minimising any negative effect. To date, most of the existing prediction methods are either lab-based or relying on specially installe…
Multi-agent deep reinforcement learning with centralized training and decentralized execution for transportation infrastructure management
organization= Department of Civil；Environmental Engineering , addressline= The Pennsylvania State University , city= University Park , postcode= 16802 , state= PA , country= USA；organization= Faculty of Architecture \& the Built Environment , addressline= Delft University of Technology , postcode= 2628 BL , state= Delft , country= The Netherlands
