离线强化学习中的分布外适应：通过因果归一化流进行反事实推理
Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows
AI总结：
针对离线强化学习的分布外适应问题，提出 MOOD-CRL，利用因果归一化流学习结构因果模型并生成反事实数据，显著优于无模型与基于模型方法。
摘要：
尽管强化学习（RL）取得了显著成功，但在线学习范式的普遍使用阻碍了其广泛应用，尤其是在危险或代价高昂的场景中。离线 RL 已成为一种替代方案，可从预先收集的静态数据集中学习。然而，这种离线学习带来了一个被称为分布偏移的新挑战：当策略在与训练数据集分布外（OOD）的场景上评估时，性能会下降。大多数现有离线 RL 通过在给定数据集所支持的信息内正则化策略学习来解决该问题。但是，此类正则化忽视了数据集之外可能存在的高回报区域的潜力。这促使人们探索新颖的离线学习技术，以便在不损害策略性能的情况下超越数据支持进行改进，可能的途径是从数据集中学习因果关系（原因与结果）而非相关性。本文提出 MOOD-CRL（基于模型的离线 OOD 适应因果 RL）算法，旨在通过因果推理而非策略正则化方法来解决离线策略训练中的外推挑战。具体而言，作者开发了因果归一化流（CNF），用于学习转移函数和奖励函数，以在离线策略评估和训练中进行数据生成与增强。基于数据不变的、基于物理的定性因果图以及观测数据，作者为 CNF 开发了一种新的学习方案，以学习定量结构因果模型。因此，CNF 获得了面向序贯决策任务的预测和反事实推理能力，展现出 OOD 适应的高潜力。作者通过实证评估验证了这种基于 CNF 的离线 RL 方法，结果表明其以显著优势优于无模型和基于模型的方法。
英文摘要：
Despite notable successes of Reinforcement Learning (RL), the prevalent use of an online learning paradigm prevents its widespread adoption, especially in hazardous or costly scenarios. Offline RL has emerged as an alternative solution, learning from pre-collected static datasets. However, this offline learning introduces a new challenge known as distributional shift, degrading the performance when the policy is evaluated on scenarios that are Out-Of-Distribution (OOD) from the training dataset. Most existing offline RL resolves this issue by regularizing policy learning within the information supported by the given dataset. However, such regularization overlooks the potential for high-reward regions that may exist beyond the dataset. This motivates exploring novel offline learning techniques that can make improvements beyond the data support without compromising policy performance, potentially by learning causation (cause-and-effect) instead of correlation from the dataset. In this paper, we propose the MOOD-CRL (Model-based Offline OOD-Adapting Causal RL) algorithm, which aims to address the challenge of extrapolation for offline policy training through causal inference instead of policy-regularizing methods. Specifically, Causal Normalizing Flow (CNF) is developed to learn the transition and reward functions for data generation and augmentation in offline policy evaluation and training. Based on the data-invariant, physics-based qualitative causal graph and the observational data, we develop a novel learning scheme for CNF to learn the quantitative structural causal model. As a result, CNF gains predictive and counterfactual reasoning capabilities for sequential decision-making tasks, revealing a high potential for OOD adaptation. Our CNF-based offline RL approach is validated through empirical evaluations, outperforming model-free and model-based methods by a significant margin.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
