同时利用离线与在线数据的无模型鲁棒 $\phi$-散度强化学习
Model-Free Robust $ϕ$-Divergence Reinforcement Learning Using Both Offline and Online Data
AI总结：
针对名义模型与真实环境不匹配导致的参数不确定性，论文提出无模型 RPQ 与混合离线/在线数据的 HyTQ 算法，并在一般函数逼近和大状态空间下给出鲁棒策略的理论保证。
摘要：
鲁棒 $\phi$-正则化马尔可夫决策过程（RRMDP）框架专注于设计控制策略，使其能够抵御因模拟器（名义）模型与真实环境之间不匹配而导致的参数不确定性。本工作作出两项重要贡献。首先，我们提出一种名为 Robust $\phi$-regularized fitted Q-iteration（RPQ，鲁棒 $\phi$-正则化拟合Q迭代）的无模型算法，用于学习 $\varepsilon$-最优鲁棒策略；该算法仅使用通过在名义模型上执行一个行为策略（具有鲁棒探索要求）所收集的历史数据。据我们所知，我们首次针对一类 $\phi$-散度给出统一分析，表明其可在具有一般函数逼近的高维系统中获得鲁棒最优策略。其次，我们引入混合鲁棒 $\phi$-正则化强化学习框架，以同时利用历史数据和在线采样来学习最优鲁棒策略。面向该框架，我们提出一种名为 Hybrid robust Total-variation-regularized Q-iteration（HyTQ，发音为 height-Q）的无模型算法。据我们所知，在混合鲁棒 $\phi$-正则化强化学习框架下，我们首次在具有一般函数逼近的大规模问题中给出了改进的分布外数据假设。最后，我们为所提算法在任意大状态空间系统上学到的策略性能提供了理论保证。
英文摘要：
The robust $ϕ$-regularized Markov Decision Process (RRMDP) framework focuses on designing control policies that are robust against parameter uncertainties due to mismatches between the simulator (nominal) model and real-world settings. This work makes two important contributions. First, we propose a model-free algorithm called Robust $ϕ$-regularized fitted Q-iteration (RPQ) for learning an $ε$-optimal robust policy that uses only the historical data collected by rolling out a behavior policy (with robust exploratory requirement) on the nominal model. To the best of our knowledge, we provide the first unified analysis for a class of $ϕ$-divergences achieving robust optimal policies in high-dimensional systems with general function approximation. Second, we introduce the hybrid robust $ϕ$-regularized reinforcement learning framework to learn an optimal robust policy using both historical data and online sampling. Towards this framework, we propose a model-free algorithm called Hybrid robust Total-variation-regularized Q-iteration (HyTQ: pronounced height-Q). To the best of our knowledge, we provide the first improved out-of-data-distribution assumption in large-scale problems with general function approximation under the hybrid robust $ϕ$-regularized reinforcement learning framework. Finally, we provide theoretical guarantees on the performance of the learned policies of our algorithms on systems with arbitrary large state space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
