arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01825stat.MEcs.LGmath.OCstat.ML
收藏

通过群鲁棒马尔可夫决策过程从多个来源学习顺序决策

Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes

Department of Statistics and Data Science, National University of Singapore(新加坡国立大学统计与数据科学系);Department of Neurology, University of Pittsburgh(匹兹堡大学神经病学系);Department of Biostatistics, Harvard T.H. Chan School of Public Health(哈佛大学T.H. Chan公共卫生学院生物统计学系);Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系);Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology(香港科学与技术大学工业工程与决策分析系)

Mingyuan Xu, Zongqi Xia, Tianxi Cai, Doudou Zhou, Nian Si

AI总结:

本文提出了一种群鲁棒马尔可夫决策过程框架,通过特征层面的不确定性集和悲观价值迭代算法,从多地点异质数据中学习鲁棒的顺序决策策略。

摘要:

我们经常从多个地点(例如医院)收集数据,这些地点具有共同的结构,但也表现出异质性。本文旨在从此类离线、多地点数据集中学习鲁棒的顺序决策制定策略。为了建模跨地点的不确定性,我们研究了具有群线性结构的分布鲁棒MDP:所有地点共享一个共同的特征映射,且转移核和预期奖励函数都在这些共享特征上是线性的。我们引入了特征层面(d-矩形)的不确定性集,这在保持可计算的鲁棒贝尔曼递归的同时,维持了关键的跨地点结构。在此基础上,我们开发了一种基于悲观价值迭代的离线算法,包括:(i)针对每个地点的岭回归用于贝尔曼目标,(ii)特征层面的最坏情况(行级最小化)聚合,以及(iii)从逆设计矩阵的对角线计算的数据依赖悲观惩罚。我们进一步提出了一个集群层面的扩展,通过利用对地点相似性的先验知识来合并相似的地点以提高样本效率。在一种鲁棒部分覆盖假设下,我们为所得到的策略证明了一个次优界。总体而言,我们的框架解决了具有异质数据源的多地点学习,并提供了一种不依赖于强状态-动作矩形性假设的鲁棒规划的系统方法。

英文摘要:

Learning a common policy from multiple source environments requires accounting for cross-source heterogeneity, and the offline setting introduces an additional challenge of limited data coverage. We formulate this problem as a distributionally robust Markov decision process under a shared linear MDP representation with source-specific transition and reward components. The uncertainty set is constructed through feature-wise mixtures of these components, yielding a $d$-rectangular structure that captures cross-source heterogeneity and admits a robust Bellman recursion. Building on this formulation, we develop Group-Robust Pessimistic Value Iteration (GR-PEVI), an offline policy-learning algorithm that combines feature-wise worst-case aggregation with a data-dependent pessimism penalty. Under a robust partial coverage condition, we establish finite-sample suboptimality guarantees for GR-PEVI and extend the analysis to clustered sources, characterizing the trade-off between the statistical gain from pooling similar sources and the approximation error induced by within-cluster heterogeneity. Simulation studies demonstrate the effectiveness of GR-PEVI, with empirical behavior consistent with the theoretical analysis. We apply GR-PEVI to a multi-source electronic health record study of multiple sclerosis treatment decisions to illustrate its use in a clinical setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑