鲁棒策略扩展：面向多样数据损坏的离线到在线强化学习
Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
AI总结：
针对离线到在线强化学习在数据损坏下性能下降的问题，提出基于逆概率加权的鲁棒策略扩展方法RPEX，在D4RL数据集上实现最先进性能。
摘要：
在离线数据上预训练策略，然后通过在线交互进行微调，即离线到在线强化学习（O2O RL），已成为现实世界强化学习部署的一种有前景的范式。然而，实际环境中的离线数据集和在线交互往往带有噪声，甚至受到恶意破坏，严重降低了O2O RL的性能。现有工作主要侧重于通过在线探索来缓解离线策略的保守性，而O2O RL在数据损坏（包括状态、动作、奖励和动力学）下的鲁棒性仍未得到探索。在本工作中，我们观察到数据损坏会导致策略出现重尾行为，从而显著降低在线探索的效率。为解决此问题，我们将逆概率加权（IPW）纳入在线探索策略以减轻重尾性，并提出了一种新颖、简单而有效的方法，称为RPEX：鲁棒策略扩展（Robust Policy Expansion）。在D4RL数据集上的大量实验结果表明，RPEX在广泛的数据损坏场景下达到了最先进的O2O性能。代码可在https://github.com/felix-thu/RPEX获取。
英文摘要：
Pretraining a policy on offline data followed by fine-tuning through online interactions, known as Offline-to-Online Reinforcement Learning (O2O RL), has emerged as a promising paradigm for real-world RL deployment. However, both offline datasets and online interactions in practical environments are often noisy or even maliciously corrupted, severely degrading the performance of O2O RL. Existing works primarily focus on mitigating the conservatism of offline policies via online exploration, while the robustness of O2O RL under data corruption, including states, actions, rewards, and dynamics, is still unexplored. In this work, we observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To address this issue, we incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness, and propose a novel, simple yet effective method termed $\textbf{RPEX}$: $\textbf{R}$obust $\textbf{P}$olicy $\textbf{EX}$pansion. Extensive experimental results on D4RL datasets demonstrate that RPEX achieves SOTA O2O performance across a wide range of data corruption scenarios. Code is available at $\href{https://github.com/felix-thu/RPEX}{https://github.com/felix-thu/RPEX}$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
