基于因果提示的模型化离线强化学习
Causal prompting model-based offline reinforcement learning
AI总结：
针对在线系统数据集高度次优且多样的挑战，提出CPRL框架，通过隐参数块因果提示动态建模环境并复用技能训练单一策略，在噪声环境中实现稳健决策并开源代码与真实医疗数据集。
摘要：
模型化离线强化学习（RL）允许智能体充分利用预收集的数据集，而无需进行额外或不道德的探索。然而，将模型化离线RL应用于在线系统面临挑战，主要原因是这些系统生成的数据集高度次优（充满噪声）且具有多样性。为解决这些问题，我们引入了因果提示强化学习（CPRL）框架，专为高度次优和资源受限的在线场景设计。CPRL的初始阶段引入了隐参数块因果提示动态（Hip-BCPD）来建模环境动态。该方法利用不变的因果提示并对齐隐参数，以泛化到新的多样化在线用户。在后续阶段，通过整合可复用技能训练单一策略来解决多任务问题，避免了从零开始训练。在包含不同噪声水平的数据集上进行的实验（包括模拟数据集和来自Dnurse APP的真实离线数据集）表明，我们提出的方法能在分布外和噪声环境中做出稳健决策，优于现有算法。此外，我们分别验证了Hip-BCPD和技能复用策略对性能鲁棒性的贡献。我们进一步分析了Hip-BCPD的可视化结构和子技能的可解释性。我们发布了源代码及首个用于精准医疗决策任务的真实医疗数据集。
英文摘要：
Model-based offline Reinforcement Learning (RL) allows agents to fully utilise pre-collected datasets without requiring additional or unethical explorations. However, applying model-based offline RL to online systems presents challenges, primarily due to the highly suboptimal (noise-filled) and diverse nature of datasets generated by online systems. To tackle these issues, we introduce the Causal Prompting Reinforcement Learning (CPRL) framework, designed for highly suboptimal and resource-constrained online scenarios. The initial phase of CPRL involves the introduction of the Hidden-Parameter Block Causal Prompting Dynamic (Hip-BCPD) to model environmental dynamics. This approach utilises invariant causal prompts and aligns hidden parameters to generalise to new and diverse online users. In the subsequent phase, a single policy is trained to address multiple tasks through the amalgamation of reusable skills, circumventing the need for training from scratch. Experiments conducted across datasets with varying levels of noise, including simulation-based and real-world offline datasets from the Dnurse APP, demonstrate that our proposed method can make robust decisions in out-of-distribution and noisy environments, outperforming contemporary algorithms. Additionally, we separately verify the contributions of Hip-BCPDs and the skill-reuse strategy to the robustness of performance. We further analyse the visualised structure of Hip-BCPD and the interpretability of sub-skills. We released our source code and the first ever real-world medical dataset for precise medical decision-making tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
