Markov switching models (MSMs) are probabilistic models that employ multiple sets of parameters to describe different dynamic regimes that a time series may exhibit at different periods of time. The switching mechanism between regimes is controlled by unobserved random variables that form a first-order Markov chain. Explicit-duration MSMs contain additional variables that explicitly model the distribution of time spent in each regime. This allows to define duration distributions of any form, but also to impose complex dependence between the observations and to reset the dynamics to initial conditions. Models that focus on the first two properties are most commonly known as hidden semi-Markov models or segment models, whilst models that focus on the third property are most commonly known as changepoint models or reset models. In this monograph, we provide a description of explicit-duration modelling by categorizing the different approaches into three groups, which differ in encoding in the explicit-duration variables different information about regime change/reset boundaries. The approaches are described using the formalism of graphical models, which allows to graphically represent and assess statistical dependence and therefore to easily describe the structure of complex models and derive inference routines. The presentation is intended to be pedagogical, focusing on providing a characterization of the three groups in terms of model structure constraints and inference properties. The monograph is supplemented with a software package that contains most of the models and examples described. The material presented should be useful to both researchers wishing to learn about these models and researchers wishing to develop them further.
A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Cost
Tsinghua University(清华大学);University of Washington(华盛顿大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
许多现实世界的应用,诸如医疗领域、推荐系统等中的应用,可以被公式化为大状态空间强化学习问题,其中仅具有策略改变数量的小预算,即,切换成本低。本文重点介绍了最近在[Yang et al 2019,Jin et al 2020]中研究的线性马尔可夫决策过程(MDP),其中线性函数近似用于大状态空间上的推广。我们提出了第一个算法的线性MDP与低开关成本。我们的算法实现了一个$\widetilde{O}\left(\sqrt{d^3H^4K}\right)$遗憾界与一个接近最优的$O\left(d H\log K\right)$全球切换成本,其中$d$是功能维度,$H$是规划视野和$K$是代理播放的剧集数。我们的遗憾界限与[Jin et al 2020]的最佳现有多项式算法相匹配,并且我们的切换成本呈指数级小于他们的。当专门用于表格MDP时,我们的转换成本界限改善了[Bai et al 2019,Zhang et al 20020]中的转换成本界限。我们补充我们的积极结果与$Ω\left(dH/\log d\right)$全球切换成本的任何无悔算法的下限。