基于线性结构 f-散度正则化的鲁棒离线强化学习
Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization
AI总结：
针对离线强化学习中的动力学偏移问题，提出线性结构 $d$-RRMDP 框架与 R2PVI 算法，利用线性函数逼近和 $f$-divergence 正则化学习鲁棒策略，并从理论上证明其近似最优且计算高效。
摘要：
鲁棒正则化马尔可夫决策过程（RRMDP）通过在值函数的转移动力学中加入正则化，来学习对动力学偏移具有鲁棒性的策略。现有方法多采用非结构化正则化，可能导致在不切实际的转移下策略过于保守。为解决此局限，我们提出新框架 $d$-矩形线性 RRMDP（$d$-RRMDP），在转移核与正则化中引入潜在结构。我们聚焦离线强化学习，智能体在名义环境中从预收集数据集学习策略。我们开发了鲁棒正则化悲观值迭代（R2PVI）算法，对转移核上基于 $f$-divergence 的正则化项，在 $d$-RRMDP 中采用线性函数逼近进行鲁棒策略学习。我们给出了 R2PVI 策略次优差距的实例依赖上界，证明这些界限受数据集对鲁棒可许转移下最优鲁棒策略访问的状态-动作空间覆盖程度的影响。我们建立信息论下界验证算法近似最优。数值实验验证 R2PVI 学得鲁棒策略，且较基线方法展现更优计算效率。
英文摘要：
The Robust Regularized Markov Decision Process (RRMDP) is proposed to learn policies robust to dynamics shifts by adding regularization to the transition dynamics in the value function. Existing methods mostly use unstructured regularization, potentially leading to conservative policies under unrealistic transitions. To address this limitation, we propose a novel framework, the $d$-rectangular linear RRMDP ($d$-RRMDP), which introduces latent structures into both transition kernels and regularization. We focus on offline reinforcement learning, where an agent learns policies from a precollected dataset in the nominal environment. We develop the Robust Regularized Pessimistic Value Iteration (R2PVI) algorithm that employs linear function approximation for robust policy learning in $d$-RRMDPs with $f$-divergence based regularization terms on transition kernels. We provide instance-dependent upper bounds on the suboptimality gap of R2PVI policies, demonstrating that these bounds are influenced by how well the dataset covers state-action spaces visited by the optimal robust policy under robustly admissible transitions. We establish information-theoretic lower bounds to verify that our algorithm is near-optimal. Finally, numerical experiments validate that R2PVI learns robust policies and exhibits superior computational efficiency compared to baseline methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
