arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.07663stat.MLcs.LG
收藏

Causal Markov Decision Processes:有效学习良好的干预措施

Causal Markov Decision Processes: Learning Good Interventions Efficiently

University of Michigan(密歇根大学);Adobe Inc.(奥多比公司)

Yangyi Lu, Amirhossein Meisami, Ambuj Tewari

摘要:

我们引入因果马尔可夫决策过程(C-MDP),一个新的形式主义顺序决策相结合的标准MDP制定因果结构的状态转移和奖励函数。许多当代和新兴的应用领域,如数字医疗和数字营销,可以受益于C-MDP建模,因为干预和状态/奖励之间的因果关系机制。我们提出了因果置信上限值迭代(C-UCBVI)算法,该算法利用C-MDP中的因果结构,并提高了不考虑因果知识的标准强化学习算法的性能。我们证明了C-UCBVI满足$\tilde{O}(HS\sqrt{ZT})$后悔界,其中$T$是总时间步,$H$是情节视界,$S$是状态空间的基数.值得注意的是,我们的后悔界限不与行动/干预的大小($A$)成比例,而只与因果图相关的量$Z$成比例,$Z$可以指数地小于$A$。通过将C-UCBVI扩展到因子化的MDP设置,我们提出了因果因子化的UCBVI(CF-UCBVI)算法,该算法进一步减少了$S$方面的指数后悔。此外,我们表明,RL算法的线性MDP问题也可以纳入C-MDP。我们的经验表明,我们的因果关系的方法在各种设置的好处,以验证我们的算法和理论结果。

英文摘要:

We introduce causal Markov Decision Processes (C-MDPs), a new formalism for sequential decision making which combines the standard MDP formulation with causal structures over state transition and reward functions. Many contemporary and emerging application areas such as digital healthcare and digital marketing can benefit from modeling with C-MDPs due to the causal mechanisms underlying the relationship between interventions and states/rewards. We propose the causal upper confidence bound value iteration (C-UCBVI) algorithm that exploits the causal structure in C-MDPs and improves the performance of standard reinforcement learning algorithms that do not take causal knowledge into account. We prove that C-UCBVI satisfies an $\tilde{O}(HS\sqrt{ZT})$ regret bound, where $T$ is the the total time steps, $H$ is the episodic horizon, and $S$ is the cardinality of the state space. Notably, our regret bound does not scale with the size of actions/interventions ($A$), but only scales with a causal graph dependent quantity $Z$ which can be exponentially smaller than $A$. By extending C-UCBVI to the factored MDP setting, we propose the causal factored UCBVI (CF-UCBVI) algorithm, which further reduces the regret exponentially in terms of $S$. Furthermore, we show that RL algorithms for linear MDP problems can also be incorporated in C-MDPs. We empirically show the benefit of our causal approaches in various settings to validate our algorithms and theoretical results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑