arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2106.07203cs.LGcs.AImath.OCstat.ML
收藏

具有通用函数逼近的强化学习在线子采样

Online Sub-Sampling for Reinforcement Learning with General Function Approximation

Peking University(北京大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Los Angles(加利福尼亚大学洛杉矶分校)

Dingwen Kong, Ruslan Salakhutdinov, Ruosong Wang, Lin F. Yang

AI总结:

本文提出在线子采样框架,通过度量数据点信息增益指导探索,将策略更新次数从Ω(K)降至poly log(K),显著降低通用函数逼近强化学习的计算复杂度。

摘要:

现有的大多数具有通用函数逼近(FA)的强化学习(RL)工作侧重于理解统计复杂度或遗憾界。然而,这类方法的计算复杂度远未被理解——事实上,在函数类上的一个简单优化问题本身可能就难以处理。在本文中,我们通过建立一个高效的在线子采样框架来解决这一问题,该框架度量由RL算法收集的数据点的信息增益,并利用该度量来指导探索。对于一个具有复杂度有界函数类的基于价值的方法,我们证明在运行RL算法K个回合时,策略仅需更新∝poly log(K)次,同时仍能获得较小的近最优遗憾界。与现有方法至少需要更新策略Ω(K)次相比,我们的方法在求解策略时大幅减少了优化调用的次数。当应用于\cite{wang2020reinforcement}或\cite{jin2021bellman}中的设置时,我们将整体时间复杂度至少提高了K倍。最后,我们通过将在线子采样技术应用于无奖励RL设置和多智能体RL设置,展示了该技术的通用性。

英文摘要:

Most of the existing works for reinforcement learning (RL) with general function approximation (FA) focus on understanding the statistical complexity or regret bounds. However, the computation complexity of such approaches is far from being understood -- indeed, a simple optimization problem over the function class might be as well intractable. In this paper, we tackle this problem by establishing an efficient online sub-sampling framework that measures the information gain of data points collected by an RL algorithm and uses the measurement to guide exploration. For a value-based method with complexity-bounded function class, we show that the policy only needs to be updated for $\propto\operatorname{poly}\log(K)$ times for running the RL algorithm for $K$ episodes while still achieving a small near-optimal regret bound. In contrast to existing approaches that update the policy for at least $Ω(K)$ times, our approach drastically reduces the number of optimization calls in solving for a policy. When applied to settings in \cite{wang2020reinforcement} or \cite{jin2021bellman}, we improve the overall time complexity by at least a factor of $K$. Finally, we show the generality of our online sub-sampling technique by applying it to the reward-free RL setting and multi-agent RL setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑