具有后验混合贝叶斯信念的正则化离线策略优化
Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
AI总结：
提出后验混合贝叶斯信念（PhyB）以统一量化离线强化学习中的认知不确定性，并基于此开发迭代正则化策略优化算法，实现单调改进直至收敛。
摘要：
离线强化学习旨在从预先收集的数据集中优化策略。该范式的一个瓶颈是管理认知不确定性，这种不确定性源于有限的数据覆盖（样本层面）以及从有限数据中识别转移动态的模糊性（模型层面）。为了统一量化这些不确定性，贝叶斯强化学习通过将动态模型视为随机变量并维护相应的信念而被提出。尽管具有理论吸引力，贝叶斯强化学习中的策略优化在计算上仍然具有挑战性，因为它需要求解带有期望的复合目标。先前的方法要么采用计算可扩展性差的基于搜索的技术，要么施加牺牲贝叶斯强化学习适应性的限制性后验假设。为了解决这些局限性，我们提出了后验混合贝叶斯信念（PhyB），它将期望重新表述为动态模型子集上的凸组合。理论分析表明，这种近似引起的目标差异是有界的。基于PhyB，我们开发了一种迭代正则化策略优化算法，该算法为单调改进直至收敛提供了与度量无关的保证。实验结果表明，PhyB在各种基准测试中达到了最先进的性能。
英文摘要：
Offline reinforcement learning (RL) aims to optimize policies from pre-collected datasets. A bottleneck of this paradigm is managing epistemic uncertainty, which arises from limited data coverage (sample-level) and the ambiguity in identifying transition dynamics from finite data (model-level). To provide a unified quantification of these uncertainties, Bayesian RL has been proposed by treating the dynamics model as a random variable and maintaining a corresponding belief. Despite its theoretical appeal, policy optimization in Bayesian RL remains computationally challenging as it requires solving composite objectives with expectations. Prior methods either employ search-based techniques with poor computational scalability or impose restrictive posterior assumptions that sacrifice the adaptability of Bayesian RL. To address these limitations, we propose Posterior Hybrid Bayesian Belief (PhyB), which reformulates the expectation as a convex combination over a subset of dynamics models. Theoretical analysis demonstrates that the objective discrepancy induced by this approximation remains bounded. Based on PhyB, we develop an iterative regularized policy optimization algorithm that provides metric-agnostic guarantees for monotonic improvement until convergence. Empirical results demonstrate that PhyB achieves state-of-the-art performance on various benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
