arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.00534stat.MLcs.LG
收藏

复合马尔可夫决策过程的迁移转移 $Q$-学习

Transition Transfer $Q$-Learning for Composite Markov Decision Processes

Princeton University(普林斯顿大学);New York University(纽约大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)

Jinhang Chai, Elynn Chen, Lin Yang

AI总结:

针对迁移强化学习,提出复合MDP框架将高维转移动力学建模为低秩共享与稀疏差异之和,并设计UCB-TQL算法,实现独立于环境维度的遗憾界,有效利用结构相似性改进单任务RL。

摘要:

为了弥合迁移强化学习(RL)中经验成功与理论理解之间的差距,我们研究了一种具有可证明性能保证的原则性方法。我们引入了一种新颖的复合 MDP 框架,其中高维转移动力学被建模为表示共享结构的低秩分量与捕获任务特定变化的稀疏分量之和。这放宽了纯低秩转移模型的常见假设,允许任务共享核心动力学但保持个体变化的更现实场景。我们引入了 UCB-TQL(Upper Confidence Bound Transfer Q-Learning,上置信界迁移 Q-学习),专为多个任务共享核心线性 MDP 动力学但沿稀疏维度发散的迁移 RL 场景而设计。当在具有充足轨迹的源任务上训练后,将 UCB-TQL 应用于目标任务时,我们实现了 $\tilde{O}(\sqrt{eH^5N})$ 的遗憾界,该界独立于环境维度进行扩展。这里,$N$ 表示目标任务中的轨迹数量,而 $e$ 量化了任务之间的稀疏差异。这一结果通过有效利用结构相似性,展示了对单任务 RL 的实质性改进。我们的理论分析为 UCB-TQL 如何同时利用共享动力学并适应任务特定变化提供了严格保证。

英文摘要:

To bridge the gap between empirical success and theoretical understanding in transfer reinforcement learning (RL), we study a principled approach with provable performance guarantees. We introduce a novel composite MDP framework where high-dimensional transition dynamics are modeled as the sum of a low-rank component representing shared structure and a sparse component capturing task-specific variations. This relaxes the common assumption of purely low-rank transition models, allowing for more realistic scenarios where tasks share core dynamics but maintain individual variations. We introduce UCB-TQL (Upper Confidence Bound Transfer Q-Learning), designed for transfer RL scenarios where multiple tasks share core linear MDP dynamics but diverge along sparse dimensions. When applying UCB-TQL to a target task after training on a source task with sufficient trajectories, we achieve a regret bound of $\tilde{O}(\sqrt{eH^5N})$ that scales independently of the ambient dimension. Here, $N$ represents the number of trajectories in the target task, while $e$ quantifies the sparse differences between tasks. This result demonstrates substantial improvement over single task RL by effectively leveraging their structural similarities. Our theoretical analysis provides rigorous guarantees for how UCB-TQL simultaneously exploits shared dynamics while adapting to task-specific variations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑