arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.00291stat.MLcs.LGmath.STstat.TH
收藏

随机梯度抽样算法的Oracle下界

Oracle Lower Bounds for Stochastic Gradient Sampling Algorithms

University of California, Berkeley(加州大学伯克利分校);Google(谷歌)

Niladri S. Chatterji, Peter L. Bartlett, Philip M. Long

摘要:

考虑了$\mathbb{R}^d$中强对数凹密度的抽样问题,证明了对对数密度的随机梯度查询次数的信息论下界.几个流行的采样算法(包括许多马尔可夫链蒙特卡罗方法)操作使用随机梯度的日志密度生成一个样本,我们的研究结果建立了所有这些算法的信息理论的限制。我们发现,对于每一个算法,存在一个良好的条件强对数凹目标密度的算法产生的点的分布将至少$\varepsilon$远离目标的全变差距离,如果梯度查询的数量小于$Ω(σ^2 d/\varepsilon^2)$,其中$σ^2 d$是随机梯度的方差。我们的下限如下相结合的想法,勒卡姆不足,经常使用的比较统计实验,随着标准的信息理论工具,用于下界贝叶斯风险函数。据我们所知,我们的结果提供了第一个非平凡的维数相关的下界这个问题。

英文摘要:

We consider the problem of sampling from a strongly log-concave density in $\mathbb{R}^d$, and prove an information theoretic lower bound on the number of stochastic gradient queries of the log density needed. Several popular sampling algorithms (including many Markov chain Monte Carlo methods) operate by using stochastic gradients of the log density to generate a sample; our results establish an information theoretic limit for all these algorithms. We show that for every algorithm, there exists a well-conditioned strongly log-concave target density for which the distribution of points generated by the algorithm would be at least $\varepsilon$ away from the target in total variation distance if the number of gradient queries is less than $Ω(σ^2 d/\varepsilon^2)$, where $σ^2 d$ is the variance of the stochastic gradient. Our lower bound follows by combining the ideas of Le Cam deficiency routinely used in the comparison of statistical experiments along with standard information theoretic tools used in lower bounding Bayes risk functions. To the best of our knowledge our results provide the first nontrivial dimension-dependent lower bound for this problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑