Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes
Department of Statistics and Data Science, National University of Singapore(新加坡国立大学统计与数据科学系);Department of Neurology, University of Pittsburgh(匹兹堡大学神经病学系);Department of Biostatistics, Harvard T.H. Chan School of Public Health(哈佛大学T.H. Chan公共卫生学院生物统计学系);Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系);Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology(香港科学与技术大学工业工程与决策分析系)
Mingyuan Xu, Zongqi Xia, Tianxi Cai, Doudou Zhou, Nian Si
Learning a common policy from multiple source environments requires accounting for cross-source heterogeneity, and the offline setting introduces an additional challenge of limited data coverage. We formulate this problem as a distributionally robust Markov decision process under a shared linear MDP representation with source-specific transition and reward components. The uncertainty set is constructed through feature-wise mixtures of these components, yielding a $d$-rectangular structure that captures cross-source heterogeneity and admits a robust Bellman recursion. Building on this formulation, we develop Group-Robust Pessimistic Value Iteration (GR-PEVI), an offline policy-learning algorithm that combines feature-wise worst-case aggregation with a data-dependent pessimism penalty. Under a robust partial coverage condition, we establish finite-sample suboptimality guarantees for GR-PEVI and extend the analysis to clustered sources, characterizing the trade-off between the statistical gain from pooling similar sources and the approximation error induced by within-cluster heterogeneity. Simulation studies demonstrate the effectiveness of GR-PEVI, with empirical behavior consistent with the theoretical analysis. We apply GR-PEVI to a multi-source electronic health record study of multiple sclerosis treatment decisions to illustrate its use in a clinical setting.
Asymptotically Optimal Sequential Testing with Markovian Data
Indian Institute of Science, Bangalore(班加罗尔印度科学学院);Indian Institute of Technology, Hyderabad(海得拉巴印度理工学院);Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 – CRIStAL(里尔大学、法国国家科学研究中心、中央里尔学院、UMR 9189 – CRIStAL)
Remembering to Be Fair: Non-Markovian Fairness in Sequential Decision Making
University of Toronto(多伦多大学);Vector Institute(向量研究所);Schwartz Reisman Institute for Technology and Society(施瓦茨·赖斯曼技术与社会研究所);University of Waterloo(滑铁卢大学)
A Reduction-based Framework for Sequential Decision Making with Delayed Feedback
Center for Data Science, Peking University(北京大学数据科学中心);Zhejiang Lab(之江实验室);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);University of Washington(华盛顿大学);University of California, Berkeley(加州大学伯克利分校)