分布鲁棒的跨动力学强化学习：线性函数逼近下的可证明效率
Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximation
AI总结：
本文首次研究带线性函数逼近的在线分布鲁棒MDP用于跨动力学强化学习，通过总变差距离设计d-矩形不确定性集消除非线性误差传播，提出DR-LSVI-UCB算法并证明其多项式次优性界与状态动作空间大小无关。
摘要：
我们研究跨动力学强化学习（RL），其中策略在源域上训练并部署到不同的目标域。我们旨在通过在线分布鲁棒马尔可夫决策过程（DRMDPs）解决这一问题，学习算法主动与源域交互，同时在源域转移核的不确定性集合内最坏可能的动力学下寻求最优性能。我们首次研究了具有函数逼近的在线DRMDPs用于跨动力学RL。我们发现DRMDPs的对偶形式即使在标称转移核为线性时也可能引入非线性，从而导致误差传播。通过使用总变差距离设计d-矩形不确定性集合，我们消除了这种额外的非线性并绕过了误差传播。随后我们引入DR-LSVI-UCB，这是第一个针对具有函数逼近的跨动力学RL的可证明高效的在线DRMDP算法，并建立了一个与状态和动作空间大小无关的多项式次优性界。我们的工作为更深入理解具有线性函数逼近的在线DRMDPs的可证明效率迈出了第一步。最后，我们通过不同的数值实验验证了DR-LSVI-UCB的性能和鲁棒性。
英文摘要：
We study off-dynamics Reinforcement Learning (RL), where the policy is trained on a source domain and deployed to a distinct target domain. We aim to solve this problem via online distributionally robust Markov decision processes (DRMDPs), where the learning algorithm actively interacts with the source domain while seeking the optimal performance under the worst possible dynamics that is within an uncertainty set of the source domain's transition kernel. We provide the first study on online DRMDPs with function approximation for off-dynamics RL. We find that DRMDPs' dual formulation can induce nonlinearity, even when the nominal transition kernel is linear, leading to error propagation. By designing a $d$-rectangular uncertainty set using the total variation distance, we remove this additional nonlinearity and bypass the error propagation. We then introduce DR-LSVI-UCB, the first provably efficient online DRMDP algorithm for off-dynamics RL with function approximation, and establish a polynomial suboptimality bound that is independent of the state and action space sizes. Our work makes the first step towards a deeper understanding of the provable efficiency of online DRMDPs with linear function approximation. Finally, we substantiate the performance and robustness of DR-LSVI-UCB through different numerical experiments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
