基于鲁棒价值感知模型学习的模型驱动离线RL
Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weighting
AI总结：
ROMI通过鲁棒价值感知模型学习和隐式可微适应加权方法，改进离线RL中的模型更新稳定性与泛化能力。
摘要：
基于模型的离线强化学习（RL）旨在通过动力学模型增强离线RL的策略探索。然而，由于不可避免的动力学模型误差，可能会发生模型利用，从而降低算法性能。对抗性模型学习提供了一个理论框架，通过求解最大化最小化形式来缓解模型利用问题。在这样的范式中，RAMBO~\citep{rigter2022rambo} 已经成为一种代表性且最受欢迎的方法，它提供了具有模型梯度的实用实现。然而，我们实证发现，在仅轻微调整超参数的情况下，RAMBO会出现严重的Q值低估和梯度爆炸，表明它倾向于过于保守，并且在模型更新上存在不稳定性。为了解决这些问题，我们提出了一种鲁棒价值感知模型学习方法，即具有隐式可微适应加权的ROMI。不同于使用模型梯度更新动力学模型，ROMI引入了一种新的鲁棒价值感知模型学习方法。该方法要求动力学模型在可调整的状态不确定性集内预测未来状态，使其接近最小Q值，从而实现可控的保守性并稳定模型更新。为进一步提高多步展开期间的分布外（OOD）泛化能力，我们提出了隐式可微适应加权，一种双层优化方案，能够自适应地实现动力学和价值感知的模型学习。在D4RL和NeoRL数据集上的实验证明，ROMI显著优于RAMBO，并在RAMBO通常表现不佳的数据集上实现了竞争或更优的性能。代码可在https://github.com/zq2r/ROMI.git获取。
英文摘要：
Model-based offline reinforcement learning (RL) aims to enhance offline RL with a dynamics model that facilitates policy exploration. However, \textit{model exploitation} could occur due to inevitable model errors, degrading algorithm performance. Adversarial model learning offers a theoretical framework to mitigate model exploitation by solving a maximin formulation. Within such a paradigm, RAMBO~\citep{rigter2022rambo} has emerged as a representative and most popular method that provides a practical implementation with model gradient. However, we empirically reveal that severe Q-value underestimation and gradient explosion can occur in RAMBO with only slight hyperparameter tuning, suggesting that it tends to be overly conservative and suffers from unstable model updates. To address these issues, we propose \textbf{RO}bust value-aware \textbf{M}odel learning with \textbf{I}mplicitly differentiable adaptive weighting (ROMI). Instead of updating the dynamics model with model gradient, ROMI introduces a novel robust value-aware model learning approach. This approach requires the dynamics model to predict future states with values close to the minimum Q-value within a scale-adjustable state uncertainty set, enabling controllable conservatism and stable model updates. To further improve out-of-distribution (OOD) generalization during multi-step rollouts, we propose implicitly differentiable adaptive weighting, a bi-level optimization scheme that adaptively achieves dynamics- and value-aware model learning. Empirical results on D4RL and NeoRL datasets show that ROMI significantly outperforms RAMBO and achieves competitive or superior performance compared to other state-of-the-art methods on datasets where RAMBO typically underperforms. Code is available at https://github.com/zq2r/ROMI.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
