Colloquially speaking, image generation models based upon diffusion processes are frequently said to exhibit "hallucinations," samples that could never occur in the training data. But where do such hallucinations come from? In this paper, we study a particular failure mode in diffusion models, which we term mode interpolation. Specifically, we find that diffusion models smoothly "interpolate" between nearby data modes in the training set, to generate samples that are completely outside the support of the original training distribution; this phenomenon leads diffusion models to generate artifacts that never existed in real data (i.e., hallucinations). We systematically study the reasons for, and the manifestation of this phenomenon. Through experiments on 1D and 2D Gaussians, we show how a discontinuous loss landscape in the diffusion model's decoder leads to a region where any smooth approximation will cause such hallucinations. Through experiments on artificial datasets with various shapes, we show how hallucination leads to the generation of combinations of shapes that never existed. Finally, we show that diffusion models in fact know when they go out of support and hallucinate. This is captured by the high variance in the trajectory of the generated sample towards the final few backward sampling process. Using a simple metric to capture this variance, we can remove over 95% of hallucinations at generation time while retaining 96% of in-support samples. We conclude our exploration by showing the implications of such hallucination (and its removal) on the collapse (and stabilization) of recursive training on synthetic data with experiments on MNIST and 2D Gaussians dataset. We release our code at https://github.com/locuslab/diffusion-model-hallucination.
RetroDiff: Retrosynthesis as Multi-stage Distribution Interpolation
Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);ByteDance Research(字节跳动研究院);Department of Computer Science, Stanford University(斯坦福大学计算机科学系)
SeisFusion: Constrained Diffusion Model with Input Guidance for 3D Seismic Data Interpolation and Reconstruction
Key Laboratory of Earth Exploration and Information Techniques of Education Ministry, College of Geophysics, Chengdu University of Technology(成都理工大学地球物理学院 教育部地球勘探与信息技术重点实验室);College of Computer Science and Cyber Security, Chengdu University of Technology(成都理工大学计算机与网络安全学院)
Tackling Structural Hallucination in Image Translation with Local Diffusion
University College London(伦敦大学学院);AstraZeneca(阿斯利康);Centre for Medical Image Computing(医学影像计算中心);Department of Medical Physics and Biomedical Engineering, University College London(伦敦大学学院医学物理与生物医学工程系);Department of Computer Science, University College London(伦敦大学学院计算机科学系);Centre for AI, DS&AI, AstraZeneca, UK(阿斯利康AI、DS&AI中心(英国))
TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
Department of Engineering Science, University of Oxford, UK(牛津大学工程科学系);Hasso Plattner Institute, University of Potsdam(波茨坦大学哈索普兰特纳研究所);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(姆罕默德·本·拉希德智能大学)
Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates
Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School(先进医学计算与分析中心,麻省总医院和哈佛医学院);Department of Industrial Engineering, University of Pittsburgh(工业工程系,匹兹堡大学)
Physics-informed Diffusion Generation for Geomagnetic Map Interpolation
State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);High-Performance Intelligent Computing Research Center for Ultra-Large Scale Graph Data(超大规模图数据高性能智能计算研究中心);School of Computer and Computing Science(计算机与计算科学学院);Hangzhou City University(杭州市大学);Zhejiang Lab(浙江实验室)
Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
Motherbrain, EQT Group(Motherbrain、EQT集团);KTH Royal Institute of Technology(皇家理工学院);Mitsubishi UFJ Trust Investment Technology Institute(三菱UFJ信托投资技术研究所);Technical University of Denmark(丹麦技术大学)
Supersampling Stable Diffusion and Beyond: Training-Free Neural Network Scaling via Signal Interpolation
School of Data and Sciences BRAC University, Dhaka(数据与科学学院,布拉克大学,达卡);Computing for Sustainability and Social Good (C2SG) Research Group Department of Computer Science and Engineering United Internation University, Dhaka(可持续性与社会公益(C2SG)研究组,计算机科学与工程系,联合国际大学,达卡)
Conditional Stochastic Interpolation for Generative Learning
Department of Applied Mathematics, The Hong Kong Polytechnic University, Hong Kong SAR, China(应用数学系,香港理工大学,香港特别行政区,中国);Department of Data Science and AI, The Hong Kong Polytechnic University, Hong Kong SAR, China(数据科学与人工智能系,香港理工大学,香港特别行政区,中国)
Lipschitz-Guided Design of Interpolation Schedules in Generative Models
Department of Mathematics, University of California, Los Angeles, CA, USA(加州大学洛杉矶分校数学系);Machine Learning Lab, Capital Fund Management, Paris, France(Capital Fund Management机器学习实验室);Courant Institute, New York University, NY, USA(纽约大学Courant研究所);Now at University of Maryland, College Park, MD, USA(现在位于马里兰大学 College Park 分校)