We study the problem of learning mixtures of $k$ Gaussians in $d$ dimensions. We make no separation assumptions on the underlying mixture components: we only require that the covariance matrices have bounded condition number and that the means and covariances lie in a ball of bounded radius. We give an algorithm that draws $d^{\mathrm{poly}(k/\varepsilon)}$ samples from the target mixture, runs in sample-polynomial time, and constructs a sampler whose output distribution is $\varepsilon$-far from the unknown mixture in total variation. Prior works for this problem either (i) required exponential runtime in the dimension $d$, (ii) placed strong assumptions on the instance (e.g., spherical covariances or clusterability), or (iii) had doubly exponential dependence on the number of components $k$.
Our approach departs from commonly used techniques for this problem like the method of moments. Instead, we leverage a recently developed reduction, based on diffusion models, from distribution learning to a supervised learning task called score matching. We give an algorithm for the latter by proving a structural result showing that the score function of a Gaussian mixture can be approximated by a piecewise-polynomial function, and there is an efficient algorithm for finding it. To our knowledge, this is the first example of diffusion models achieving a state-of-the-art theoretical guarantee for an unsupervised learning task.
Score-of-Mixture Training: Training One-Step Generative Models Made Simple via Score Estimation of Mixture Distributions
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology (MIT), Cambridge, Massachusetts, USA(电气工程与计算机科学系,麻省理工学院(MIT),剑桥,马萨诸塞州,美国)
提出Score-of-Mixture Training (SMT)框架,通过最小化$α$-skew Jensen-Shannon散度并估计混合分布得分来训练单步生成模型,支持从零开始训练与预训练扩散模型蒸馏,实验证明其在CIFAR-10和ImageNet 64x64上表现优异。
Learning Energy-Based Models in High-Dimensional Spaces with Multi-scale Denoising Score Matching
Redwood Center for Theoretical Neuroscience(雷德伍德理论神经科学中心);University of California Berkeley(加州大学伯克利分校);Berkeley AI Research(伯克利人工智能研究院);Helen Wills Neuroscience Institute(海伦·威尔斯神经科学研究所)
An Explicit Surrogate for Gaussian Mixture Flow Matching with Wasserstein Gap Bounds
Université Paris-Saclay, Inria, CIAMS(巴黎-萨克雷大学, 法国国家信息与自动化研究所, CIAMS);Department of Electrical Engineering and Computer Science, University of Quebec in Trois-Rivieres (UQTR)(魁北克大学三河城分校电气工程与计算机科学系)
Shortest-Path Flow Matching with Mixture-Conditioned Bases for OOD Generalization to Unseen Conditions
Wellcome Sanger Institute(Wellcome桑格研究所);Institute for Molecular Medicine Finland (FIMM)(芬兰分子医学研究所);University of Helsinki(赫尔辛基大学);School of Computing and Information Systems(计算与信息系统学院);University of Cambridge(剑桥大学);Cambridge Centre for AI in Medicine(剑桥人工智能医学中心);MediaTek Research(MediaTek研究);Cambridge Stem Cell Institute(剑桥干细胞研究所)
Soft Diffusion: Score Matching for General Corruptions
Google Research(谷歌研究院);UT Austin(得克萨斯大学奥斯汀分校)
我们定义了一个更广泛的家庭腐败过程,概括了以前已知的扩散模型。为了扭转这些一般的扩散,我们提出了一个新的目标,称为软分数匹配,可证明学习任何线性腐败过程的分数函数,并为CelebA产生最先进的结果。软分数匹配结合了网络中的退化过程。我们的新损失训练模型预测一个干净的图像,\textit{that after corruption},匹配扩散观察。我们表明,我们的目标学习梯度的可能性在适当的规律性条件下的一个家庭的腐败过程。我们进一步开发了一种原则性的方法来选择一般扩散过程的腐败水平和一种新的采样方法,我们称之为动量采样器。我们的实验表明,我们的框架适用于一般的线性腐败过程,如高斯模糊和掩蔽。我们在CelebA-64上实现了最先进的FID评分$1.85$,优于所有先前的线性扩散模型。与香草去噪扩散相比,我们还显示出显着的计算优势。
Simplifying Flow Matching Transformations with Low-Rank Mixture Models
Stanford Intelligent Systems Laboratory in the Department of Aeronautics and Astronautics at Stanford University(斯坦福大学航空航天系智能系统实验室);Industrial and Systems Engineering Department at King Fahd University of Petroleum and Minerals(科威特石油矿物大学工业与系统工程系)
Quantum Dynamics via Score Matching on Bohmian Trajectories
Beijing National Laboratory for Condensed Matter Physics(北京凝聚态物理实验室);Institute of Physics, Chinese Academy of Sciences, Beijing 100190, China(中国科学院物理研究所,北京100190,中国)