arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.04132cs.LG
收藏

线性Bandits和线性RL的近优表示学习

Near-optimal Representation Learning for Linear Bandits and Linear RL

Peking University(北京大学);Princeton University(普林斯顿大学);Amazon(亚马逊);Beijing Institute of Big Data Research(北京大数据研究院)

Jiachen Hu, Xiaoyu Chen, Chi Jin, Lihong Li, Liwei Wang

摘要:

研究了多任务线性强盗和多任务情景强化学习中线性值函数逼近的表示学习。我们首先考虑这样的设置,其中我们同时使用$M$线性强盗和维度$d$,并且这些强盗共享公共的$k$维度线性表示,使得$k\ll d$和$k \ll M$。我们提出了一个样本效率的算法,MTLR-OFUL,它利用共享表示实现$\tilde{O}(M\sqrt{dkT} + d\sqrt{kMT} )$遗憾,$T$的总步骤数。我们的遗憾显着改善了通过独立解决每个任务而实现的基线$\tilde{O}(Md\sqrt{T})$。我们进一步发展了一个下限,表明我们的遗憾是接近最佳时,$d > M$。此外,我们扩展的算法和分析,以多任务情景RL与线性值函数近似下的低固有贝尔曼误差\citep{zanette2020learning}。据我们所知,这是第一个理论结果,它描述了多任务表示学习在RL中探索函数逼近的好处。

英文摘要:

This paper studies representation learning for multi-task linear bandits and multi-task episodic RL with linear value function approximation. We first consider the setting where we play $M$ linear bandits with dimension $d$ concurrently, and these bandits share a common $k$-dimensional linear representation so that $k\ll d$ and $k \ll M$. We propose a sample-efficient algorithm, MTLR-OFUL, which leverages the shared representation to achieve $\tilde{O}(M\sqrt{dkT} + d\sqrt{kMT} )$ regret, with $T$ being the number of total steps. Our regret significantly improves upon the baseline $\tilde{O}(Md\sqrt{T})$ achieved by solving each task independently. We further develop a lower bound that shows our regret is near-optimal when $d > M$. Furthermore, we extend the algorithm and analysis to multi-task episodic RL with linear value function approximation under low inherent Bellman error \citep{zanette2020learning}. To the best of our knowledge, this is the first theoretical result that characterizes the benefits of multi-task representation learning for exploration in RL with function approximation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑