We identify and analyze a surprising phenomenon of Latent Diffusion Models (LDMs) where the final steps of the diffusion can degrade sample quality. In contrast to conventional arguments that justify early stopping for numerical stability, this phenomenon is intrinsic to the dimensionality reduction in LDMs. We provide a principled explanation by analyzing the interaction between latent dimension and stopping time. Under a Gaussian framework with linear autoencoders, we characterize the conditions under which early stopping is needed to minimize the distance between generated and target distributions. More precisely, we show that lower-dimensional representations benefit from earlier termination, whereas higher-dimensional latent spaces require later stopping time. We further establish that the latent dimension interplays with other hyperparameters of the problem such as constraints in the parameters of score matching. Experiments on synthetic and real datasets illustrate these properties, underlining that early stopping can improve generative quality. Together, our results offer a theoretical foundation for understanding how the latent dimension influences the sample quality, and highlight stopping time as a key hyperparameter in LDMs.
Inria(法国国家信息与自动化技术研究院);PSL Research University(巴黎社会科学高等研究院);CMAP(巴黎高等理工学院应用数学与计算科学实验室);Ecole Polytechnique Palaiseau, France(法国巴黎高等理工学院Palaiseau分校);Sakana AI Tokyo, Japan(日本东京Sakana AI公司)
Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation
Polymathic AI;Flatiron Institute;University of Liège(列日大学);New York University(纽约大学);Princeton University(普林斯顿大学);Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM(巴黎-萨克莱大学、巴黎-cite大学、CEA、CNRS、AIM)
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
UCLA(加州大学洛杉矶分校);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院);Salesforce Research(赛富时研究院);Google Brain(谷歌大脑);Peking University(北京大学);Tsinghua University(清华大学)
3M-Diffusion: Latent Multi-Modal Diffusion for Language-Guided Molecular Structure Generation
Artificial Intelligence Research Laboratory(人工智能研究实验室);College of Information Sciences and Technology(信息科学与技术学院);Center for Artificial Intelligence Foundations and Scientific Applications(人工智能基础与科学应用中心);Institute for Computational and Data Sciences(计算与数据科学研究所);The Pennsylvania State University(宾夕法尼亚州立大学)
Disentangling Continuous-Time Latent Dynamics: Identifiability of Latent SDEs via Diffusion Shifts
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);The University of Melbourne(墨尔本大学);Peking University(北京大学);Carnegie Mellon University(卡内基梅隆大学)