眼见不为实:针对虚假相关性的鲁棒强化学习
Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation
AI总结:
本文提出鲁棒状态混杂马尔可夫决策过程(RSC-MDPs),从理论和算法层面解决强化学习中由未观测混杂因素导致的虚假相关性问题,并在自动驾驶与操作任务中验证其优于现有鲁棒方法。
摘要:
在强化学习(RL)中,鲁棒性已被广泛研究,以应对各种形式的不确定性,如随机扰动、罕见事件和恶意攻击。本文考虑一种关键的鲁棒性类型——针对虚假相关性的鲁棒性,即状态的不同部分之间不存在由未观测混杂因素诱导的相关性。这些虚假相关性在现实世界任务中无处不在,例如,由于不可观测的人类活动,自动驾驶汽车通常在白天观察到交通拥堵,而在夜间观察到交通畅通。一个学习了这种无用甚至有害相关性的模型,当测试场景中的混杂因素偏离训练场景时,可能会灾难性地失效。尽管动机明确,但实现针对虚假相关性的鲁棒性面临重大挑战,因为由未观测混杂因素和因果结构所塑造的不确定性集合难以刻画和识别。因此,现有假设简单且无结构不确定性集合的鲁棒算法不足以应对这一挑战。为解决该问题,我们提出了鲁棒状态混杂马尔可夫决策过程(RSC-MDPs),并从理论上证明了与其他鲁棒RL方法相比,其在避免学习虚假相关性方面的优越性。我们还设计了一种经验算法来学习RSC-MDPs的鲁棒最优策略,该策略在八个真实的自动驾驶和操作任务中优于所有基线方法。
英文摘要:
Robustness has been extensively studied in reinforcement learning (RL) to handle various forms of uncertainty such as random perturbations, rare events, and malicious attacks. In this work, we consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. These spurious correlations are ubiquitous in real-world tasks, for instance, a self-driving car usually observes heavy traffic in the daytime and light traffic at night due to unobservable human activity. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Although motivated, enabling robustness against spurious correlation poses significant challenges since the uncertainty set, shaped by the unobserved confounder and causal structure, is difficult to characterize and identify. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge. To solve this issue, we propose Robust State-Confounded Markov Decision Processes (RSC-MDPs) and theoretically demonstrate its superiority in avoiding learning spurious correlations compared with other robust RL counterparts. We also design an empirical algorithm to learn the robust optimal policy for RSC-MDPs, which outperforms all baselines in eight realistic self-driving and manipulation tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。