arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.00726cs.LGmath.STstat.MLstat.TH
收藏

从等视野法看分数匹配的统计效率

Statistical Efficiency of Score Matching: The View from Isoperimetry

Stanford University(斯坦福大学);Carnegie Mellon University(卡内基梅隆大学)

Frederic Koehler, Alexander Heckett, Andrej Risteski

摘要:

参数化到归一化常数的深度生成模型(例如,基于能量的模型)难以通过最大化数据的似然性来训练,因为其似然性和/或梯度不能被明确地或有效地写下。分数匹配是一种训练方法,我们不是拟合训练数据的似然$\log p(x)$,而是拟合分数函数$\nabla_x \log p(x)$ --避免了评估分区函数的需要。虽然这个估计量已知是一致的,但它的统计效率是否(以及何时)与最大似然法相当还不清楚,最大似然法已知是(渐近)最优的。我们在本文中启动这一调查线,并显示统计效率的得分匹配和等周属性的分布估计-即庞加莱,对数-Sobolev和等周常数-数量管理的混合时间的马尔可夫过程,如朗之万动力学之间的紧密联系。粗略地说,我们表明,分数匹配估计是统计上可比的最大似然分布时,有一个小的等周常数。相反,如果分布有一个大的等周常数-即使是简单的分布族,如指数族,具有足够丰富的统计数据-分数匹配将大大低于最大似然法的效率。我们适当地形式化这些结果在有限样本制度,并在渐近制度。最后,我们确定了一个直接平行的离散设置,在那里我们连接的统计性质的pseudolikkastrointestimation近似张量熵和Glauber动力学。

英文摘要:

Deep generative models parametrized up to a normalizing constant (e.g. energy-based models) are difficult to train by maximizing the likelihood of the data because the likelihood and/or gradients thereof cannot be explicitly or efficiently written down. Score matching is a training method, whereby instead of fitting the likelihood $\log p(x)$ for the training data, we instead fit the score function $\nabla_x \log p(x)$ -- obviating the need to evaluate the partition function. Though this estimator is known to be consistent, its unclear whether (and when) its statistical efficiency is comparable to that of maximum likelihood -- which is known to be (asymptotically) optimal. We initiate this line of inquiry in this paper, and show a tight connection between statistical efficiency of score matching and the isoperimetric properties of the distribution being estimated -- i.e. the Poincaré, log-Sobolev and isoperimetric constant -- quantities which govern the mixing time of Markov processes like Langevin dynamics. Roughly, we show that the score matching estimator is statistically comparable to the maximum likelihood when the distribution has a small isoperimetric constant. Conversely, if the distribution has a large isoperimetric constant -- even for simple families of distributions like exponential families with rich enough sufficient statistics -- score matching will be substantially less efficient than maximum likelihood. We suitably formalize these results both in the finite sample regime, and in the asymptotic regime. Finally, we identify a direct parallel in the discrete setting, where we connect the statistical properties of pseudolikelihood estimation with approximate tensorization of entropy and the Glauber dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑