Score-based diffusion models have achieved incredible performance in generating realistic images, audio, and video data. While these models produce high-quality samples with impressive details, they often introduce unrealistic artifacts, such as distorted fingers or hallucinated texts with no meaning. This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner. Through experimental probing, we consistently observe that such phenomenon is attributed it to the network's local generation bias. Denoising networks tend to produce outputs that rely heavily on highly correlated local regions, particularly when different dimensions of the data distribution are nearly pairwise independent. This behavior leads to a generation process that decomposes the global distribution into separate, independent distributions for each symbol, ultimately failing to capture the global structure, including underlying grammar. Intriguingly, this bias persists across various denoising network architectures including MLP and transformers which have the structure to model global dependency. These findings also provide insights into understanding other types of hallucinations, extending beyond text, as a result of implicit biases in the denoising models. Additionally, we theoretically analyze the training dynamics for a specific case involving a two-layer MLP learning parity points on a hypercube, offering an explanation of its underlying mechanism.
How I Met Your Bias: Investigating Bias Amplification in Diffusion Models
LTCI, Télécom Paris, Institut Polytechnique de Paris(LTCI、 Télécom Paris、 Institut Polytechnique de Paris);MaLGa-DIBRIS, University of Genova(MaLGa-DIBRIS、乌尔比诺大学);AIGO, Istituto Italiano di Tecnologia(AIGO、意大利技术研究所)
Tackling Structural Hallucination in Image Translation with Local Diffusion
University College London(伦敦大学学院);AstraZeneca(阿斯利康);Centre for Medical Image Computing(医学影像计算中心);Department of Medical Physics and Biomedical Engineering, University College London(伦敦大学学院医学物理与生物医学工程系);Department of Computer Science, University College London(伦敦大学学院计算机科学系);Centre for AI, DS&AI, AstraZeneca, UK(阿斯利康AI、DS&AI中心(英国))
TraceDet: Hallucination Detection from the Decoding Trace of Diffusion Large Language Models
Department of Engineering Science, University of Oxford, UK(牛津大学工程科学系);Hasso Plattner Institute, University of Potsdam(波茨坦大学哈索普兰特纳研究所);Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(姆罕默德·本·拉希德智能大学)
Hallucination-Aware Diffusion Sampling for Inverse Problems via Robust Prior Updates
Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School(先进医学计算与分析中心,麻省总医院和哈佛医学院);Department of Industrial Engineering, University of Pittsburgh(工业工程系,匹兹堡大学)