The main goal of this paper is to apply the so-called policy iteration algorithm (PIA) for the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. In order to do that we first derive some important properties for a pseudo-Poisson equation associated to the problem. In the sequence it is shown that the convergence of the PIA to a solution satisfying the optimality equation holds under some classical hypotheses and that this optimal solution yields to an optimal control strategy for the average control problem for the continuous-time PDMP in a feedback form.
In this paper we aim to construct infinite dimensional versions of well established Piecewise Deterministic Monte Carlo methods, such as the Bouncy Particle Sampler, the Zig-Zag Sampler and the Boomerang Sampler. In order to do so we provide an abstract infinite-dimensional framework for Piecewise Deterministic Markov Processes (PDMPs) with unbounded event intensities. We further develop exponential convergence to equilibrium of the infinite dimensional Boomerang Sampler, using hypocoercivity techniques. Furthermore we establish how the infinite dimensional Boomerang Sampler admits a finite di…
Our paper illustrates how the theory of Lie systems allows recovering known results and provide new examples of piecewise deterministic processes with phase-type jumps for which the corresponding first-time passage problems may be solved explicitly.
This paper deals with the general discounted impulse control problem of a piecewise deterministic Markov process. We investigate a new family of epsilon-optimal strategies. The construction of such strategies is explicit and only necessitates the previous knowledge of the cost of the no-impulse strategy. In particular, it does not require the resolution of auxiliary optimal stopping problem or the computation of the value function at each point of the state space. This approach is based on the iteration of a single-jump-or-intervention operator associated to the piecewise deterministic Markov …
We consider piecewise deterministic Markov processes with degenerate transition kernels of the "house-of-cards"-type. We use a splitting scheme based on jump times to prove the absolute continuity, as well as some regularity, of the invariant measure of the process. Finally, we obtain finer results on the regularity of the one-dimensional marginals of the invariant measure, using integration by parts with respect to the jump times.
We study the time reversal of a general PDMP. The time reversed process is defined as $X_{(T-t)-}$, where $T$ is some given time and $X_t$ is a stationary PDMP. We obtain the parameters of the reversed process, like the jump intensity and the jump measure.
We analyze the asymptotics of crossing a high piecewise linear barriers by a renewal compound process with the subexponential jumps. The study is motivated by ruin probabilities of two insurance companies (or two branches of the same company) that divide between them both claims and premia in some specified proportions when the initial reserves of both companies tend to infinity.
The purpose of this paper is to study a Markovian metapopulation model on a directed graph with edge-supported transfers and deterministic intra-nodal population dynamics. We first state tractable stability conditions for two typical frameworks motivated by applications: constant jump rates with multiplicative transfer amplitudes, and coercive jump rates with unitary transfers. More general criteria for boundedness, petiteness and ergodicity are then given.
We study the asymptotic behavior of multiscale stochastic gene networks using weak limits of Markov jump processes. Depending on the time and concentration scales of the system we distinguish four types of limits: continuous piecewise deterministic processes (PDP) with switching, PDP with jumps in the continuous variables, averaged PDP, and PDP with singular switching. We justify rigorously the convergence for the four types of limits. The convergence results can be used to simplify the stochastic dynamics of gene network models arising in molecular biology.
This paper deals with the optimal stopping problem under partial observation for piecewise-deterministic Markov processes. We first obtain a recursive formulation of the optimal filter process and derive the dynamic programming equation of the partially observed optimal stopping problem. Then, we propose a numerical method, based on the quantization of the discrete-time filter process and the inter-jump times, to approximate the value function and to compute an actual $ε$-optimal stopping time. We prove the convergence of the algorithms and bound the rates of convergence.
In this paper, we are interested in the exact simulation of a class of Piecewise Deterministic Markov Processes (PDMP). We show how to perform efficient thinning algorithms depending on the jump rate bound. For different types of jump rate bounds, we compare theoretically the mean number of generated (total) jump times and we compare numerically the simulation times. We use thinning algorithms on Hodgkin-Huxley models with Markovian ion channels dynamic to illustrate the results.
Given an energy potential on the Euclidian space, a piecewise deterministic Markov process is designed to sample the corresponding Gibbs measure. In dimension one an Eyring-Kramers formula is obtained for the exit time of the domain of a local minimum at low temperature, and a necessary and sufficient condition is given on the cooling schedule in a simulated annealing algorithm to ensure the process converges to the set of global minima. This condition is similar to the classical one for diffusions and involves the critical depth of the potential. In higher dimension a non optimal sufficient c…
This paper presents a numerical method to calculate the value function for a general discounted impulse control problem for piecewise deterministic Markov processes. Our approach is based on a quantization technique for the underlying Markov chain defined by the post jump location and inter-arrival time. Convergence results are obtained and more importantly we are able to give a convergence rate of the algorithm. The paper is illustrated by a numerical example.
We present a numerical method to compute the survival function and the moments of the exit time for a piecewise-deterministic Markov process (PDMP). Our approach is based on the quantization of an underlying discrete-time Markov chain related to the PDMP. The approximation we propose is easily computable and is even flexible with respect to the exit time we consider. We prove the convergence of the algorithm and obtain bounds for the rate of convergence in the case of the moments. An academic example and a model from the reliability field illustrate the paper.
In [20], the authors addressed the question of the averaging of a slow-fast Piecewise Deterministic Markov Process (PDMP) in infinite dimension. In the present paper, we carry on and complete this work by the mathematical analysis of the fluctuation of the slow-fast system around the averaged limit. A central limit theorem is derived and the associated Langevin approximation is considered. The motivation of this work is a stochastic Hodgkin-Huxley model which describes the propagation of an action potential along the nerve fiber. We study this PDMP in detail and provide more general results fo…
In the first part of this paper we study approximations of trajectories of Piecewise Deter-ministic Processes (PDP) when the flow is not explicit by the thinning method. We also establish a strong error estimate for PDPs as well as a weak error expansion for Piecewise Deterministic Markov Processes (PDMP). These estimates are the building blocks of the Multilevel Monte Carlo (MLMC) method which we study in the second part. The coupling required by MLMC is based on the thinning procedure. In the third part we apply these results to a 2-dimensional Morris-Lecar model with stochastic ion channels…
This paper investigates the random horizon optimal stopping problem for measure-valued piecewise deterministic Markov processes (PDMPs). This is motivated by population dynamics applications, when one wants to monitor some characteristics of the individuals in a small population. The population and its individual characteristics can be represented by a point measure. We first define a PDMP on a space of locally finite measures. Then we define a sequence of random horizon optimal stopping problems for such processes. We prove that the value function of the problems can be obtained by iterating …
In this paper we define an infinite-dimensional controlled piecewise deterministic Markov process (PDMP) and we study an optimal control problem with finite time horizon and unbounded cost. This process is a coupling between a continuous time Markov Chain and a set of semilinear parabolic partial differential equations, both processes depending on the control. We apply dynamic programming to the embedded Markov decision process to obtain existence of optimal relaxed controls and we give some sufficient conditions ensuring the existence of an optimal ordinary control. This study, which constitu…
We consider a general piecewise deterministic Markov process (PDMP) $X=\{X_t\}_{t\geqslant 0}$ with measure-valued generator $\mathcal{A}$, for which the conditional distribution function of the inter-occurrence time is not necessarily absolutely continuous. A general form of the exponential martingales is presented as $$M^f_t=\frac{f(X_t)}{f(X_0)}\left[\mathrm{Sexp}\left(\int_{(0,t]}\frac{\mathrm{d}L(\mathcal{A}f)_s}{f(X_{s-})}\right)\right]^{-1}.$$ Using this exponential martingale as a likelihood ratio process, we define a new probability measure. It is shown that the original process remai…
The main goal of this paper is to derive sufficient conditions for the existence of an optimal control strategy for the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. In order to do that we apply the so-called vanishing discount approach to obtain a solution to an average cost optimality inequality associated to the long run average cost problem. Our main assumptions are written in terms of some integro-differential inequalities related to the …
In this paper, we consider the generalized Hodgkin-Huxley model introduced by Austin in \cite{Austin}. This model describes the propagation of an action potential along the axon of a neuron at the scale of ion channels. Mathematically, this model is a fully-coupled Piecewise Deterministic Markov Process (PDMP) in infinite dimension. We introduce two time scales in this model in considering that some ion channels open and close at faster jump rates than others. We perform a slow-fast analysis of this model and prove that asymptotically this two time scales model reduces to the so called average…
We propose a numerical method to approximate the value function for the optimal stopping problem of a piecewise deterministic Markov process (PDMP). Our approach is based on quantization of the post jump location---inter-arrival time Markov chain naturally embedded in the PDMP, and path-adapted time discretization grids. It allows us to derive bounds for the convergence rate of the algorithm and to provide a computable $ε$-optimal stopping time. The paper is illustrated by a numerical example.
We consider multiscale stochastic spatial gene networks involving chemical reactions and diffusions. The model is Markovian and the transitions are driven by Poisson random clocks. We consider a case where there are two different spatial scales: a microscopic one with fast dynamic and a macroscopic one with slow dynamic. At the microscopic level, the species are abundant and for the large population limit a partial differential equation (PDE) is obtained. On the contrary at the macroscopic level, the species are not abundant and their dynamic remains governed by jump processes. It results that…
In Turitsyn, Chertkov, Vucelja (2011) a non-reversible Markov Chain Monte Carlo (MCMC) method on an augmented state space was introduced, here referred to as Lifted Metropolis-Hastings (LMH). A scaling limit of the magnetization process in the Curie-Weiss model is derived for LMH, as well as for Metropolis-Hastings (MH). The required jump rate in the high (supercritical) temperature regime equals $n^{1/2}$ for LMH, which should be compared to $n$ for MH. At the critical temperature the required jump rate equals $n^{3/4}$ for LMH and $n^{3/2}$ for MH, in agreement with experimental results of T…
We consider a piecewise-deterministic Markov process governed by a jump intensity function, a rate function that determines the behaviour between jumps, and a stochastic kernel describing the conditional distribution of jump sizes. We study the point process of upcrossings of a level $b$ by the Markov process. Our main result shows that, under a suitable scaling $ν(b)$, the point process converges, as $b$ tends to infinity, weakly to a geometrically compound Poisson process. We also prove a version of Rice's formula relating the stationary density of the process to level crossing intensities. …
A piecewise-deterministic Markov process, specified by random jumps and switching semi-flows, as well as the associated Markov chain given by its post-jump locations, are investigated in this paper. The existence of an exponentially attracting invariant measure and the strong law of large numbers are proven for the chain. Further, a one-to-one correspondence between invariant measures for the chain and invariant measures for the continuous-time process is established. This result, together with the aforementioned ergodic properties of the discrete-time model, is used to derive the strong law o…
Let g(x)=x/2 + 17/30 (mod 1), let ξ_i, i= 1,2,... be a sequence of independent, identically distributed random variables with uniform distribution on the interval [0,1/15], define g_i(x)=g(x)+ ξ_i (mod 1) and, for n=1,2,..., define g^n(x)=g_n(g_{n-1}(...(g_1(x))...)). For x \in [0,1) let μ_{n,x} denote the distribution of g^n(x). The purpose of this note is to show that there exists a unique probability measure μ, such that, for all x \in [0,1), μ_{n,x} tends to μ, as n tends to infinity. This contradicts a claim by Lasota and Mackey from 1987 stating that the process has an asymptotic three-p…
We investigate a piecewise-deterministic Markov process, evolving on a Polish metric space, whose deterministic behaviour between random jumps is governed by some semi-flow, and any state right after the jump is attained by a randomly selected continuous transformation. It is assumed that the jumps appear at random moments, which coincide with the jump times of a Poisson process with intensity $λ$. The model of this type, although in a more general version, was examined in our previous papers, where we have shown, among others, that the Markov process under consideration possesses a unique inv…
Necessary and sufficient conditions are given for a substochastic semigroup on $L^1$ obtained through the Kato--Voigt perturbation theorem to be either stochastic or strongly stable. We show how such semigroups are related to piecewise deterministic Markov process, provide a probabilistic interpretation of our results, and apply them to fragmentation equations.
We present limit theorems for a sequence of Piecewise Deterministic Markov Processes (PDMPs) taking values in a separable Hilbert space. This class of processes provides a rigorous framework for stochastic spatial models in which discrete random events are globally coupled with continuous space-dependent variables solving partial differential equations, e.g., stochastic hybrid models of excitable membranes. We derive a law of large numbers which establishes a connection to deterministic macroscopic models and a martingale central limit theorem which connects the stochastic fluctuations to diff…
We investigate piecewise deterministic Markov processes (PDMP), where the deterministic dynamics follows a scalar conservation law and random jumps in the system are characterized by changes in the flux function. We show under which assumptions we can guarantee the existence of a PDMP and conclude bounded variation estimates for sample paths. Finally, we apply this dynamics to a production and traffic model and use this framework to incorporate the well-known scattering of flux functions observed in data sets.
Piecewise deterministic Markov processes are an important new tool in the design of Markov Chain Monte Carlo algorithms. Two examples of fundamental importance are the Bouncy Particle Sampler (BPS) and the Zig-Zag process (ZZ). In this paper scaling limits for both algorithms are determined. Here the dimensionality of the space tends towards infinity and the target distribution is the multivariate standard normal distribution. For several quantities of interest (angular momentum, first coordinate, and negative log-density) the scaling limits show qualitatively very different and rich behaviour…
For a class of piecewise deterministic Markov processes, the supports of the invariant measures are characterized. This is based on the analysis of controllability properties of an associated deterministic control system. Its invariant control sets determine the supports.
We consider a piecewise-deterministic Markov process (PDMP) with general conditional distribution of inter-occurrence time, which is called a general PDMP here. Our purpose is to establish the theory of measure-valued generator for general PDMPs. The additive functional of a semi-dynamic system (SDS) is introduced firstly, which presents us an analytic tool for the whole paper. The additive functionals of a general PDMP are represented in terms of additive functionals of the SDS. The necessary and sufficient conditions of being a local martingale or a special semimartingale for them are given.…
This paper deals with the long run average continuous control problem of piecewise deterministic Markov processes (PDMP's) taking values in a general Borel space and with compact action space depending on the state variable. The control variable acts on the jump rate and transition measure of the PDMP, and the running and boundary costs are assumed to be positive but not necessarily bounded. Our first main result is to obtain an optimality equation for the long run average cost in terms of a discrete-time optimality equation related to the embedded Markov chain given by the post-jump location …
We present a numerical method to compute expectations of functionals of a piecewise-deterministic Markov process. We discuss time dependent functionals as well as deterministic time horizon problems. Our approach is based on the quantization of an underlying discrete-time Markov chain. We obtain bounds for the rate of convergence of the algorithm. The approximation we propose is easily computable and is flexible with respect to some of the parameters defining the problem. Two examples illustrate the paper.
We present a short introduction into the framework of piecewise deterministic Markov processes. We illustrate the abstract mathematical setting with a series of examples related to dispersal of biological systems, cell cycle models, gene expression, physiologically structured populations, as well as neural activity. General results concerning asymptotic properties of stochastic semigroups induced by such Markov processes are applied to specific examples.
The law of the iterated logarithm for a piecewise deterministic Markov process assured by the properties of the Markov chain given by its post-jump locations
We aim at characterizing viability, invariance and some reachability properties of controlled piecewise deterministic Markov processes (PDMPs). Using analytical methods from the theory of viscosity solutions, we establish criteria for viability and invariance in terms of the first order normal cone. We also investigate reachability of arbitrary open sets. The method is based on viscosity techniques and duality for some associated linearized problem. The theoretical results are applied to general On/Off systems, Cook's model for haploinssuficiency, and a stochastic model for bacteriophage lambd…
In this paper, we consider a piecewise deterministic Markov process (PDMP), with known flow and deterministic transition measure, and unknown jump rate $λ$. To estimate nonparametrically the jump rate, we first construct an adaptive estimator of the stationary density, then we derive a quotient estimator $\hatλ_n$ of $λ$. We provide uniform bounds for the risk of these estimators, and prove that the estimator of the jump rate is nearly minimax (up to a $\ln^2(n)$ factor). Simulations illustrate the behavior of our estimator.
A classical random walk $(S_t, t\in\mathbb{N})$ is defined by $S_t:=\displaystyle\sum_{n=0}^t X_n$, where $(X_n)$ are i.i.d. When the increments $(X_n)_{n\in\mathbb{N}}$ are a one-order Markov chain, a short memory is introduced in the dynamics of $(S_t)$. This so-called "persistent" random walk is nolonger Markovian and, under suitable conditions, the rescaled process converges towards the integrated telegraph noise (ITN) as the time-scale and space-scale parameters tend to zero (see Herrmann and Vallois, 2010; Tapiero-Vallois, Tapiero-Vallois2}). The ITN process is effectively non-Markovian …
We extend the hypocoercivity framework for piecewise-deterministic Markov process (PDMP) Monte Carlo established in [Andrieu et. al. (2018)] to heavy-tailed target distributions, which exhibit subgeometric rates of convergence to equilibrium. We make use of weak Poincaré inequalities, as developed in the work of [Grothaus and Wang (2019)], the ideas of which we adapt to the PDMPs of interest. On the way we report largely potential-independent approaches to bounding explicitly solutions of the Poisson equation of the Langevin diffusion and its first and second derivatives, required here to cont…
Piecewise Deterministic Markov Processes (PDMPs) are studied in a general framework. First, different constructions are proven to be equivalent. Second, we introduce a coupling between two PDMPs following the same differential flow which implies quantitative bounds on the total variation between the marginal distributions of the two processes. Finally two results are established regarding the invariant measures of PDMPs. A practical condition to show that a probability measure is invariant for the associated PDMP semi-group is presented. In a second time, a bound on the invariant probability m…
It is shown that piecewise deterministic dissipative quantum dynamics in a vector space with indefinite metric can lead to well defined, positive probabilities. The case of quantum jumps on the Poincar'e disk is studied in details, including results of numerical simulations of quantum fractals. Keywords: iterated function systems, dynamical semigroups, EEQT, indefinite metric, quantum jumps, quantum measurement, non-commuting observables, fractals, Poincare disk
We establish $L^2$-exponential convergence rate for three popular piecewise deterministic Markov processes for sampling: the randomized Hamiltonian Monte Carlo method, the zigzag process, and the bouncy particle sampler. Our analysis is based on a variational framework for hypocoercivity, which combines a Poincaré-type inequality in time-augmented state space and a standard $L^2$ energy estimate. Our analysis provides explicit convergence rate estimates, which are more quantitative than existing results.
In this paper, a class of piecewise deterministic Markov processes with underlying fast dynamic is studied. Using a "penalty method" , an averaging result is obtained when the underlying dynamic is infinitely accelerated. The features of the averaged process, which is still a piecewise deterministic Markov process, are fully described.
In this work, we give sharp asymptotic equivalents in the small temperature regime of the smallest eigenvalues of the generator of some piecewise deterministic Markov processes (including the ZigZag process and the Bouncy Particle Sampler process) with refreshment rate $α$ on the one-dimensional torus T. These asymptotic equivalents are usually called Eyring-Kramers type formulas in the literature. The case when the refreshment rate $α$ vanishes on T is also considered.
We present recent results on Piecewise Deterministic Markov Processes (PDMPs), involved in biological modeling. PDMPs, first introduced in the probabilistic literature by Davis (1984), are a very general class of Markov processes and are being increasingly popular in biological applications. They also give new interesting challenges from the theoretical point of view. We give here different examples on the long time behavior of switching Markov models applied to population dynamics, on uniform sampling in general branching models applied to structured population dynamic, on time scale separati…
The Bouncy Particle sampler (BPS) and the Zig-Zag sampler (ZZS) are continuous time, non-reversible Monte Carlo methods based on piecewise deterministic Markov processes. Experiments show that the speed of convergence of these samplers can be affected by the shape of the target distribution, as for instance in the case of anisotropic targets. We propose an adaptive scheme that iteratively learns all or part of the covariance matrix of the target and takes advantage of the obtained information to modify the underlying process with the aim of increasing the speed of convergence. Moreover, we def…
In this paper, we analyse piecewise deterministic Markov processes, as introduced in Davis (1984). Many models in insurance mathematics can be formulated in terms of the general concept of piecewise deterministic Markov processes. In this context, one is interested in computing certain quantities of interest such as the probability of ruin of an insurance company, or the insurance company's value, defined as the expected discounted future dividend payments until the time of ruin. Instead of explicitly solving the integro-(partial) differential equation related to the quantity of interest consi…
We give a short overview of recent results on a specific class of Markov process: the Piecewise Deterministic Markov Processes (PDMPs). We first recall the definition of these processes and give some general results. On more specific cases such as the TCP model or a model of switched vector fields, better results can be proved, especially as regards long time behaviour. We continue our review with an infinite dimensional example of neuronal activity. From the statistical point of view, these models provide specific challenges: we illustrate this point with the example of the estimation of the …
Exponential ergodicity in the bounded-Lipschitz distance for a subclass of piecewise-deterministic Markov processes with random switching between flows
In piecewise-deterministic Markov processes (PDMPs) the state of a finite-dimensional system evolves continuously, but the evolutive equation may change randomly as a result of discrete switches. A running cost is integrated along the corresponding piecewise-deterministic trajectory up to the termination to produce the cumulative cost of the process. We address three natural questions related to uncertainty in cumulative cost of PDMP models: (1) how to compute the Cumulative Distribution Function (CDF) of the cumulative cost when the switching rates are fully known; (2) how to accurately bound…
