Reflect-then-Plan：通过双贝叶斯视角进行基于模型的离线规划
Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens
AI总结：
本文提出双贝叶斯离线基于模型规划方法 RefPlan，将规划视为贝叶斯后验估计并在部署时利用实时观测更新环境动态信念，从而提升保守离线 RL 策略在有限数据、高不确定性和动态变化环境中的性能与鲁棒性。
摘要：
当在线探索代价高昂或不安全时，离线强化学习（RL）至关重要，但由于数据有限，它往往难以应对较高的认知不确定性。现有方法依赖固定的保守策略，限制了适应性和泛化能力。为解决这一问题，我们提出 Reflect-then-Plan（RefPlan），这是一种新颖的双贝叶斯离线基于模型（MB）规划方法。RefPlan 通过将规划重新表述为贝叶斯后验估计，统一了不确定性建模与 MB 规划。在部署阶段，它利用实时观测更新关于环境动态的信念，并通过边缘化将不确定性纳入 MB 规划。在标准基准测试上的实验结果表明，RefPlan 显著提升了保守离线 RL 策略的性能。特别是，RefPlan 在高认知不确定性和数据有限的情况下保持了稳健性能，同时展现出对不断变化的环境动态的韧性，提升了离线学习策略的灵活性、泛化性和鲁棒性。
英文摘要：
Offline reinforcement learning (RL) is crucial when online exploration is costly or unsafe but often struggles with high epistemic uncertainty due to limited data. Existing methods rely on fixed conservative policies, restricting adaptivity and generalization. To address this, we propose Reflect-then-Plan (RefPlan), a novel doubly Bayesian offline model-based (MB) planning approach. RefPlan unifies uncertainty modeling and MB planning by recasting planning as Bayesian posterior estimation. At deployment, it updates a belief over environment dynamics using real-time observations, incorporating uncertainty into MB planning via marginalization. Empirical results on standard benchmarks show that RefPlan significantly improves the performance of conservative offline RL policies. In particular, RefPlan maintains robust performance under high epistemic uncertainty and limited data, while demonstrating resilience to changing environment dynamics, improving the flexibility, generalizability, and robustness of offline-learned policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
