Diffusion models are capable of generating photo-realistic images that combine elements which likely do not appear together in the training set, demonstrating the ability to \textit{compositionally generalize}. Nonetheless, the precise mechanism of compositionality and how it is acquired through training remains elusive. Inspired by cognitive neuroscientific approaches, we consider a highly reduced setting to examine whether and when diffusion models learn semantically meaningful and factorized representations of composable features. We performed extensive controlled experiments on conditional Denoising Diffusion Probabilistic Models (DDPMs) trained to generate various forms of 2D Gaussian bump images. We found that the models learn factorized but not fully continuous manifold representations for encoding continuous features of variation underlying the data. With such representations, models demonstrate superior feature compositionality but limited ability to interpolate over unseen values of a given feature. Our experimental results further demonstrate that diffusion models can attain compositionality with few compositional examples, suggesting a more efficient way to train DDPMs. Finally, we connect manifold formation in diffusion models to percolation theory in physics, offering insight into the sudden onset of factorized representation learning. Our thorough toy experiments thus contribute a deeper understanding of how diffusion models capture compositional structure in data.
NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models
Microsoft Research(微软研究院);Microsoft Azure(微软Azure);University of Science and Technology of China(中国科学技术大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Zhejiang University(浙江大学);The University of Tokyo(东京大学);Peking University(北京大学)
MSF: Efficient Diffusion Model Via Multi-Scale Latent Factorize
Huawei Inc.(华为公司);Huazhong University of Science & Technology(华中科技大学);The Chinese University of Hong Kong(香港中文大学);School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院)
Unsupervised Decomposition and Recombination with Discriminator-Driven Diffusion Models
Research Laboratory of Electronics, MIT, Cambridge, MA 02139, USA(MIT电子研究实验室);Department of Physics, Massachusetts Institute of Technology, MIT, Cambridge, MA, USA(麻省理工学院物理系);NSF AI Institute for Artificial Intelligence(国家科学基金会人工智能与基本相互作用研究所);Kempner Institute, Harvard University, Cambridge, MA, USA(哈佛大学 Kempner 院);Google DeepMind, Mountain View, CA, USA(Google DeepMind);School of Computer Science, Georgia Institute of Technology, Atlanta, USA(佐治亚理工学院计算机科学系)
Compose Your Policies! Improving Diffusion-based or Flow-based Robot Policies via Test-time Distribution-level Composition
The University of Hong Kong(香港大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Shanghai AI Lab(上海人工智能实验室);Shanghai Jiaotong University(上海交通大学);The Hong Kong University of Science and Technology(香港科技大学)
CDMT-EHR: A Continuous-Time Diffusion Framework for Generating Mixed-Type Time-Series Electronic Health Records
Department of Core Informatics, Graduate School of Informatics, Osaka Metropolitan University(核心信息学系,信息学研究生学校,大阪 Metropolitan 大学);Smart Data & Knowledge Services Department, German Research Center for Artificial Intelligence (DFKI GmbH)(智能数据与知识服务部,德国人工智能研究中心(DFKI GmbH));Department of Hematology and Department of Laboratory Medicine and Medical Informatics, Graduate School of Medicine, Osaka Metropolitan University(血液科和实验室医学与医学信息学系,医学研究生学校,大阪 Metropolitan 大学)
Flexible Multitask Learning with Factorized Diffusion Policy
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Harvard University(哈佛大学);Norwegian University of Science and Technology(挪威科学与技术大学);Columbia University(哥伦比亚大学)
A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
Institute of Physics, Ecole Polytechnique Fédérale de Lausanne(洛桑联邦理工学院物理研究所);Institute of Electrical and Micro Engineering, Ecole Polytechnique Fédérale de Lausanne(洛桑联邦理工学院电气与微工程研究所)
RNDiff: Rainfall nowcasting with Condition Diffusion Model
Faculty of Artificial Intelligence and Big Data(人工智能与大数据学院);Chongqing University of Technology(重庆理工大学);Yibin University(宜春大学);Chengdu University of Information Technology(成都信息工程大学)
Variational Bayesian Flow Network for Graph Generation
School of Computer Science, Wuhan University, Wuhan, China(武汉大学计算机学院);Department of Computing, Macquarie University, Sydney, Australia(麦觉奎大学计算系);Griffith University, Gold Coast, Australia(格里菲斯大学);Wuhan University Shenzhen Research Institute, Shenzhen, China(武汉大学深圳研究院)
Disentangled representations via score-based variational autoencoders
Center for Neural Science, New York University(纽约大学神经科学中心);Center for Computational Neuroscience, Flatiron Institute(Flatiron研究所计算神经科学中心);Center for Data Science, New York University(纽约大学数据科学中心)
LILAC: Layer-Wise Independent LoRAs and Cascaded Conditioning for Multi-Concept Customization of Diffusion Models
Adobe Research, Romania(罗马尼亚Adobe研究院);Department of Computer Science, University of Bucharest, Romania(罗马尼亚布加勒斯特大学计算机科学系);International Computer High School of Bucharest, Romania(罗马尼亚布加勒斯特国际计算机高中)