Agnostic Q-learning with Function Approximation in Deterministic Systems: Tight Bounds on Approximation Error and Sample Complexity
Institute for Advanced Study（高等研究院）；Princeton University（普林斯顿大学）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Carnegie Mellon University（卡内基梅隆大学）
Simon S. Du, Jason D. Lee, Gaurav Mahajan, Ruosong Wang
本文研究确定性系统中具有函数逼近的不可知$Q$学习问题，其中最优$Q$函数可由类$\mathcal{F}$中的函数逼近，逼近误差为$δ\ge 0$ .我们提出了一种新的基于递归的算法，并表明，如果$δ= O\left(ρ/\sqrt{\dim_E}\right)$，那么可以找到最佳的政策使用$O\left(\dim_E\right)$轨迹，其中$ρ$是最佳的$Q$ -值的最佳行动和第二最佳行动和$\dim_E$之间的差距是$\mathcal{F}$的Eluder维。我们的结果有两个含义：1）结合[Du et al.，ICLR 2020]，我们的上界表明条件$δ= \widetildeΘ\left(ρ/\sqrt{\mathrm{dim}_E}\right)$对于多项式样本复杂度的算法是必要和充分的. 2)结合[Wen and Van Roy，NIPS 2013]中的下限，我们的上限表明，即使在不可知论的设置中，样本复杂度$\widetildeΘ\left(\mathrm{dim}_E\right)$也是紧密的。因此，我们解决了[Wen and Van Roy，NIPS 2013]中提出的不可知$Q$学习的开放问题。我们进一步扩展我们的算法的随机奖励设置，并获得类似的结果。
英文摘要：
The current paper studies the problem of agnostic $Q$-learning with function approximation in deterministic systems where the optimal $Q$-function is approximable by a function in the class $\mathcal{F}$ with approximation error $δ\ge 0$. We propose a novel recursion-based algorithm and show that if $δ= O\left(ρ/\sqrt{\dim_E}\right)$, then one can find the optimal policy using $O\left(\dim_E\right)$ trajectories, where $ρ$ is the gap between the optimal $Q$-value of the best actions and that of the second-best actions and $\dim_E$ is the Eluder dimension of $\mathcal{F}$. Our result has two implications:
1) In conjunction with the lower bound in [Du et al., ICLR 2020], our upper bound suggests that the condition $δ= \widetildeΘ\left(ρ/\sqrt{\mathrm{dim}_E}\right)$ is necessary and sufficient for algorithms with polynomial sample complexity.
2) In conjunction with the lower bound in [Wen and Van Roy, NIPS 2013], our upper bound suggests that the sample complexity $\widetildeΘ\left(\mathrm{dim}_E\right)$ is tight even in the agnostic setting.
Therefore, we settle the open problem on agnostic $Q$-learning proposed in [Wen and Van Roy, NIPS 2013]. We further extend our algorithm to the stochastic reward setting and obtain similar results.
Deterministic Exploration via Stationary Bellman Error Maximization
Center for Artificial Intelligence and Data Science, University of Würzburg（维尔茨堡大学人工智能与数据科学中心）；Department of Computer Science, Technical University of Darmstadt（达姆施塔特工业大学计算机科学系）；Hessian Center for Artificial Intelligence（黑森州人工智能中心）
Tight Bounds on $\ell_1$ Approximation and Learning of Self-Bounding Functions
Google Brain（谷歌大脑）；Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）
We study the complexity of learning and approximation of self-bounding functions over the uniform distribution on the Boolean hypercube ${0,1}^n$ . Informally, a function $f:{0,1}^n \rightarrow \mathbb{R}$ is self-bounding if for every $x \in {0,1}^n$ , $f(x)$ upper bounds the sum of all the $n$ marginal decreases in the value of the function at $x$ . Self-bounding functions include such well-known classes of functions as submodular and fractionally-subadditive (XOS) functions. They were introduced by Boucheron et al. (2000) in the context of concentration of measure inequalities. Our main res…
Nearly Tight Bounds for Robust Proper Learning of Halfspaces with a Margin
University of Wisconsin, Madison（威斯康星大学麦迪逊分校）；University of California, San Diego（加利福尼亚大学圣地亚哥分校）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Google Research（谷歌研究院）
Tighter Generalization Bounds for Iterative Differentially Private Learning Algorithms
UBTECH Sydney AI Centre, School of Computer Science, Faculty of Engineering, the University of Sydney（悉尼大学工程学院计算机科学学院优必选悉尼人工智能中心）；University of Science and Technology of China（中国科学技术大学）
Agnostic Learning under Targeted Poisoning: Optimal Rates and the Role of Randomness
Department of Mathematics, Technion（技术学院数学系）；Departments of Mathematics, Computer Science, and Data and Decision Sciences, Technion and Google Research（技术学院数学系、计算机科学系和数据与决策科学系，以及谷歌研究）
