通向可靠策略的剪枝之路：一种面向重症监护的多目标深度 Q-Learning 方法
Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care
AI总结：
针对医疗决策中稀疏奖励导致离线强化学习不稳定的问题，提出一种多目标深度Q-learning方法，通过动作剪枝整合频繁测量的生物标志物信号，在重症监护场景下显著提升了策略的可靠性。
摘要：
医疗治疗通常涉及一系列决策，每个决策都基于先前的结果。这一过程与强化学习（RL）高度契合，后者是一个在未知动态下优化序列决策以最大化累积奖励的框架。尽管 RL 在创建数据驱动的治疗方案方面展现出潜力，但由于经常需要使用主要基于死亡结果定义的稀疏奖励，其在医疗环境中的应用充满挑战。这种稀疏性会降低离线估计的稳定性，构成了充分利用 RL 进行医疗决策的重大障碍。我们引入了一种深度 Q-learning 方法，通过将相关但带噪声的频繁测量生物标志物信号整合到奖励规范中，在不影响主要结果优化的情况下，获得更可靠的重症监护策略。我们的方法在基于所有可用奖励对动作空间进行剪枝后，再在稀疏的主奖励上训练最终模型。该方法在从中间信号中提取有价值信息以指导学习的同时，最大限度地减少了主要目标的潜在失真。我们使用模拟环境和来自重症监护病房的真实健康记录，在 off-policy 和 offline 设置下评估了我们的方法。我们的实证结果表明，该方法优于常见的 offline RL 方法，如 conservative Q-learning 和 batch-constrained deep Q-learning。通过动作剪枝解耦稀疏奖励和频繁测量的奖励代理，我们的工作代表着向在数据密集型重症监护环境中有效利用大量可用信息开发可靠策略迈出了一步。
英文摘要：
Medical treatments often involve a sequence of decisions, each informed by previous outcomes. This process closely aligns with reinforcement learning (RL), a framework for optimizing sequential decisions to maximize cumulative rewards under unknown dynamics. While RL shows promise for creating data-driven treatment plans, its application in medical contexts is challenging due to the frequent need to use sparse rewards, primarily defined based on mortality outcomes. This sparsity can reduce the stability of offline estimates, posing a significant hurdle in fully utilizing RL for medical decision-making. We introduce a deep Q-learning approach to obtain more reliable critical care policies by integrating relevant but noisy frequently measured biomarker signals into the reward specification without compromising the optimization of the main outcome. Our method prunes the action space based on all available rewards before training a final model on the sparse main reward. This approach minimizes potential distortions of the main objective while extracting valuable information from intermediate signals to guide learning. We evaluate our method in off-policy and offline settings using simulated environments and real health records from intensive care units. Our empirical results demonstrate that our method outperforms common offline RL methods such as conservative Q-learning and batch-constrained deep Q-learning. By disentangling sparse rewards and frequently measured reward proxies through action pruning, our work represents a step towards developing reliable policies that effectively harness the wealth of available information in data-intensive critical care environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
