强化学习（Reinforcement Learning, RL）算法往往难以学习到能泛化到新情境的策略，原因包括因果混淆、对无关因素过拟合，以及无法隔离对状态因素的控制等。这些问题源于一个共同根源：未能准确识别并利用环境中特定于状态的因果关系。虽然部分强化学习领域的先前研究旨在明确识别这些关系，但它们依赖于非正式的特定领域启发式方法，例如空间和时间邻近性。实际因果性为确定特定事件的原因提供了一个有原则的通用框架。然而，现有的实际原因定义通常会将因果关系归因于大量事件，即便其中许多事件很少影响结果。先前关于实际因果性的研究提出将常态性作为该问题的解决方案，但现有实现难以扩展到复杂的连续值强化学习环境。本文提出了功能性实际原因（functional actual cause, FAC）框架，该框架利用环境中特定于上下文的独立性来限制实际原因的集合。我们还提出了实际原因推断联合优化（Joint Optimization for Actual Cause Inference, JACI）算法，该算法从观测数据中学习以推断功能性实际原因。我们通过实验证明，FAC在一组来自实际因果性文献的示例上与已知结果一致，且在一组复杂的连续值环境中，JACI识别实际原因的准确率显著高于现有启发式方法。
英文摘要：
Reinforcement learning (RL) algorithms often struggle to learn policies that generalize to novel situations due to issues such as causal confusion, overfitting to irrelevant factors, and failure to isolate control of state factors. These issues stem from a common source: a failure to accurately identify and exploit state-specific causal relationships in the environment. While some prior works in RL aim to identify these relationships explicitly, they rely on informal domain-specific heuristics such as spatial and temporal proximity. Actual causality offers a principled and general framework for determining the causes of particular events. However, existing definitions of actual cause often attribute causality to a large number of events, even if many of them rarely influence the outcome. Prior work on actual causality proposes normality as a solution to this problem, but its existing implementations are challenging to scale to complex and continuous-valued RL environments. This paper introduces functional actual cause (FAC), a framework that uses context-specific independencies in the environment to restrict the set of actual causes. We additionally introduce Joint Optimization for Actual Cause Inference (JACI), an algorithm that learns from observational data to infer functional actual causes. We demonstrate empirically that FAC agrees with known results on a suite of examples from the actual causality literature, and JACI identifies actual causes with significantly higher accuracy than existing heuristic methods in a set of complex, continuous-valued environments.
State Key Laboratory for Novel Software Technology \& School of Artificial Intelligence, Nanjing University, China School of Informatics, The University of Edinburgh, UK
