arXiv 2310.129552024-03-09cs.LGcs.AI

面向多样化数据损坏下的鲁棒离线强化学习

Towards Robust Offline Reinforcement Learning under Diverse Data Corruption

HKUST（香港科技大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Washington University (St. Louis)（圣路易斯华盛顿大学）；Tencent Robotics X（腾讯Robotics X实验室）；Peking University（北京大学）；CUHK (SZ)（香港中文大学（深圳））

Rui Yang, Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, Tong Zhang

备注：: Accepted by ICLR 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究针对离线强化学习易受数据损坏影响的问题，发现隐式Q学习（IQL）鲁棒性较强，通过引入Huber损失和分位数估计器改进IQL，提出了更鲁棒的RIQL算法，在多种数据损坏场景下表现优异。

摘要：

离线强化学习（RL）提供了一种从离线数据集中学习强化策略的可行方法，无需与环境进行成本高昂或不安全的交互。然而，人类在真实环境中收集的数据集通常存在噪声，甚至可能遭到恶意篡改，这会严重降低离线RL的性能。在本研究中，我们首先探究了当前离线RL算法在全面数据损坏（包括状态、动作、奖励和动力学损坏）下的性能。大量实验表明，在各类离线RL算法中，隐式Q学习（IQL，implicit Q-learning）展现出了出色的抗数据损坏能力。此外，我们开展了实证与理论双重分析以理解IQL的鲁棒性能，确认其有监督策略学习机制是关键因素。尽管IQL相对鲁棒，但在动力学损坏下，它仍会受到Q函数重尾目标的困扰。为解决这一问题，我们受鲁棒统计学启发，采用Huber损失处理重尾特性，并利用分位数估计器平衡对损坏数据的惩罚与学习稳定性。通过将这些简单却有效的改进融入IQL，我们提出了一种更具鲁棒性的离线RL方法，名为Robust IQL（RIQL）。大量实验证明，RIQL在面临多样化数据损坏场景时表现出极强的鲁棒性能。

英文摘要：

Offline reinforcement learning (RL) presents a promising approach for learning reinforced policies from offline datasets without the need for costly or unsafe interactions with the environment. However, datasets collected by humans in real-world environments are often noisy and may even be maliciously corrupted, which can significantly degrade the performance of offline RL. In this work, we first investigate the performance of current offline RL algorithms under comprehensive data corruption, including states, actions, rewards, and dynamics. Our extensive experiments reveal that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. Furthermore, we conduct both empirical and theoretical analyses to understand IQL's robust performance, identifying its supervised policy learning scheme as the key factor. Despite its relative robustness, IQL still suffers from heavy-tail targets of Q functions under dynamics corruption. To tackle this challenge, we draw inspiration from robust statistics to employ the Huber loss to handle the heavy-tailedness and utilize quantile estimators to balance penalization for corrupted data and learning stability. By incorporating these simple yet effective modifications into IQL, we propose a more robust offline RL approach named Robust IQL (RIQL). Extensive experiments demonstrate that RIQL exhibits highly robust performance when subjected to diverse data corruption scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。