Piecewise-deterministic Markov processes (PDMPs) are often used to model abrupt changes in the global environment or capabilities of a controlled system. This is typically done by considering a set of "operating modes" (each with its own system dynamics and performance metrics) and assuming that the mode can switch stochastically while the system state evolves. Such models have a broad range of applications in engineering, economics, manufacturing, robotics, and biological sciences. Here, we introduce and analyze an "occasionally observed" version of mode-switching PDMPs. We show how such systems can be controlled optimally if the planner is not alerted to mode-switches as they occur but may instead have access to infrequent mode observations. We first develop a general framework for handling this through dynamic programming on a higher-dimensional mode-belief space. While quite general, this method is rarely practical due to the curse of dimensionality. We then discuss assumptions that allow for solving the same problem much more efficiently, with the computational costs growing linearly (rather than exponentially) with the number of modes. We use this approach to derive Hamilton-Jacobi-Bellman PDEs and quasi-variational inequalities encoding the optimal behavior for a variety of planning horizons (fixed, infinite, indefinite, random) and mode-observation schemes (at fixed times or on-demand). We discuss the computational challenges associated with each version and illustrate the resulting methods on test problems from surveillance-evading path planning. We also include an example based on robotic navigation: a Mars rover that minimizes the expected time to target while accounting for the possibility of unobserved/incremental damages and dynamics-altering breakdowns.
Optimal control of infinite-dimensional Piecewise Deterministic Markov Processes: a BSDE approach. Application to the control of an excitable cell membrane
A Piecewise Deterministic Markov Toy Model for Traffic/Maintenance and Associated Hamilton-Jacobi Integrodifferential Systems on Networks
我们研究了无限时域中的最优控制问题，当动态属于一类特定的分段确定性马尔可夫过程约束的星形网络（受交通模型的启发）。我们采用[H. M. Soner具有状态空间约束的最优控制。二. SIAM J. Control Optim.，24（6）：1110.1122，1986]证明了值函数的正则性和动态规划原理。扩展网络和Krylov的“摇动系数”方法，我们证明了值函数可以被看作是一个线性化的最优化问题集的解决方案上的一组方便的概率测度。该方法完全依赖于粘度参数。作为一个副产品，对偶制定保证值函数是点态上确界相关的Hamilton-Jacobi积分微分系统的正则子解。这确保值函数满足Perron对黏性解的（唯一）候选的preconization。最后，我们证明了同样的线性化可以通过结合经典（无约束）问题的线性化和成本惩罚。后一种方法适用于非常一般的近可行系统（可能没有进一步的可控性）和不连续成本。
