各类研究中的一个主要目标是利用观测数据集，提供一套能够带来因果改进的新反事实指南。动态治疗方案（Dynamic Treatment Regimes，DTRs）被广泛用于将这一过程形式化。然而，寻找最优DTRs的现有方法往往依赖于在现实应用（例如医疗决策或公共政策）中会被违反的假设，尤其是当（a）未观测混杂因素的存在不可忽视，以及（b）未观测混杂因素随时间变化（例如受到先前行动影响）时。当此类假设被违反时，人们通常会面临关于潜在因果模型的模糊性。这种模糊性是不可避免的，因为无法从观测数据中理解未观测混杂因素的动态及其对数据观测部分的因果影响。受一项案例研究的启发，即为合作医院中接受移植并出现移植后新发糖尿病（New Onset Diabetes After Transplantation，NODAT）这一医学状况的患者寻找更优治疗方案，我们将DTRs扩展为一类称为模糊动态治疗方案（Ambiguous Dynamic Treatment Regimes，ADTRs）的新类别，其中治疗方案的因果影响基于因果模型的“云”进行评估。随后，我们将ADTRs与模糊部分可观测Mark决策过程（Ambiguous Partially Observable Mark Decision Processes，APOMDPs）联系起来，并开发强化学习方法，从而能够利用观测数据高效学习最优治疗方案。我们建立了这些学习方法的理论结果，包括（弱）一致性和渐近正态性。我们还在案例研究和仿真实验中进一步评估了这些学习方法的性能。
英文摘要：
A main research goal in various studies is to use an observational data set and provide a new set of counterfactual guidelines that can yield causal improvements. Dynamic Treatment Regimes (DTRs) are widely studied to formalize this process. However, available methods in finding optimal DTRs often rely on assumptions that are violated in real-world applications (e.g., medical decision-making or public policy), especially when (a) the existence of unobserved confounders cannot be ignored, and (b) the unobserved confounders are time-varying (e.g., affected by previous actions). When such assumptions are violated, one often faces ambiguity regarding the underlying causal model. This ambiguity is inevitable, since the dynamics of unobserved confounders and their causal impact on the observed part of the data cannot be understood from the observed data. Motivated by a case study of finding superior treatment regimes for patients who underwent transplantation in our partner hospital and faced a medical condition known as New Onset Diabetes After Transplantation (NODAT), we extend DTRs to a new class termed Ambiguous Dynamic Treatment Regimes (ADTRs), in which the causal impact of treatment regimes is evaluated based on a "cloud" of causal models. We then connect ADTRs to Ambiguous Partially Observable Mark Decision Processes (APOMDPs) and develop Reinforcement Learning methods, which enable using the observed data to efficiently learn an optimal treatment regime. We establish theoretical results for these learning methods, including (weak) consistency and asymptotic normality. We further evaluate the performance of these learning methods both in our case study and in simulation experiments.
CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Beijing Key Laboratory of Embodied Intelligence Systems, Department of Automation, Tsinghua University, Beijing, China（北京智能体智能系统重点实验室，自动化系，清华大学，北京，中国）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China（复杂系统认知与决策智能重点实验室，自动化研究所，中国科学院，北京，中国）
Patient level simulation and reinforcement learning to discover novel strategies for treating ovarian cancer
Vilcek Institute of Graduate Biomedical Sciences, NYU Grossman School of Medicine（纽约大学格罗斯曼医学院维尔切克生物医学科学研究生院）；NYU Langone Health System（纽约大学朗格尼医疗系统）；Merck & Co., Inc（默克公司）；New York University（纽约大学）
Challenges for Reinforcement Learning in Healthcare
University of Toronto（多伦多大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Li Ka Shing Centre for Healthcare Analytics Research and Training（李嘉诚医疗保健分析研究培训中心）；Unity Health Toronto（多伦多联合健康中心）；Sinai Health System（西奈健康系统）
Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh School of Medicine（匹兹堡大学医学院）；Carnegie Mellon University（卡内基梅隆大学）；McGowan Institute for Regenerative Medicine（麦高恩再生医学研究所）
A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning
Tokyo University of Agriculture and Technology（东京农业大学）；Institute of Science Tokyo（东京科学研究院）；National University of Singapore（国立新加坡大学）；LY Corporation（LY公司）；Altos Labs, Inc.（Altos实验室）；National Institute of Advanced Industrial Science and Technology (AIST)（国家先进工业科学与技术研究院）；Emory University（埃默里大学）；Norwegian University of Science and Technology（挪威科学技术大学）
Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework
London School of Economics and Political Science（伦敦政治经济学院）；Institute of Systems Science, Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院系统科学研究所）；ByteDance（字节跳动）；The Univeristy of North Carolina at Chapell Hill（北卡罗来纳大学教堂山分校）；University of Michigan（密歇根大学）；North Carolina State University（北卡罗来纳州立大学）
Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry
Automated Treatment Planning for Interstitial HDR Brachytherapy for Locally Advanced Cervical Cancer using Deep Reinforcement Learning
Department of Radiation & Cellular Oncology, University of Chicago（芝加哥大学辐射与细胞肿瘤学系）；Department of Physics, University of Chicago（芝加哥大学物理系）；Department of Radiation Oncology, Emory University（埃默里大学放射肿瘤学系）
Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number
Università “Sapienza”（罗马第一大学）；University of Rome Tor Vergata（罗马第二大学）；Istituto dei Sistemi Complessi, CNR（意大利国家研究委员会复杂系统研究所）；The Abdus Salam International Centre for Theoretical Physics - ICTP（阿卜杜勒·萨拉姆国际理论物理中心）
