扩散模型的高斯普适性
Gaussian Universality for Diffusion Models
AI总结:
本文研究扩散生成数据的高斯普适性,证明线性分类器测试误差仅由一阶、二阶统计量决定,并给出条件扩散样本的 Lipschitz 函数集中结论。
摘要:
我们研究由扩散模型生成的数据分布的高斯普适性。所谓高斯普适性,是指在扩散数据上为分类任务训练的广义线性模型 $f(\mathbf{W})$ 的测试误差,与在各类别均值和协方差相匹配的高斯混合模型上训练的 $f(\mathbf{W})$ 的测试误差一致。换言之,在线性设定下,测试误差仅取决于扩散生成数据的一阶和二阶统计量。作为推论,线性分类器测试误差的分析可从扩散生成数据归约为高斯数据。由于诸如 \cite{sehwag2024stretchingdollardiffusiontraining} 等方法的兴起,分析在合成数据上训练的模型性能是一个切题问题。此外,我们证明,对于任意 1-Lipschitz 标量函数 $ϕ$,当 $\mathbf{x}$ 从对应每个类别的条件扩散模型中采样时,$ϕ(\mathbf{x})$ 以高概率接近 $\mathbb{E} ϕ(\mathbf{x})$。最后,我们指出,当前用于证明普适性的方法不适用于扩散生成数据,因为数据协方差矩阵的最小奇异值往往趋于零,这与文献中的假设相反。这使得扩展以往的数学普适性结果成为一个引人关注的开放问题。
英文摘要:
We investigate Gaussian Universality for data distributions generated via diffusion models. By Gaussian Universality we mean that the test error of a generalized linear model $f(\mathbf{W})$ trained for a classification task on the diffusion data matches the test error of $f(\mathbf{W})$ trained on the Gaussian Mixture with matching means and covariances per class.In other words, the test error depends only on the first and second order statistics of the diffusion-generated data in the linear setting. As a corollary, the analysis of the test error for linear classifiers can be reduced to Gaussian data from diffusion-generated data. Analysing the performance of models trained on synthetic data is a pertinent problem due to the surge of methods such as \cite{sehwag2024stretchingdollardiffusiontraining}. Moreover, we show that, for any $1$- Lipschitz scalar function $ϕ$, $ϕ(\mathbf{x})$ is close to $\mathbb{E} ϕ(\mathbf{x})$ with high probability for $\mathbf{x}$ sampled from the conditional diffusion model corresponding to each class. Finally, we note that current approaches for proving universality do not apply to diffusion-generated data as the covariance matrices of the data tend to have vanishing minimum singular values, contrary to the assumption made in the literature. This leaves extending previous mathematical universality results as an intriguing open question.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。