arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.03595cs.LGcs.AI
收藏

通过局部生成偏置理解扩散模型的文本幻觉

Towards Understanding Text Hallucination of Diffusion Models via Local Generation Bias

Tsinghua University(清华大学);Princeton University(普林斯顿大学);UC Berkeley(加州大学伯克利分校);Qiuzhen College, Tsinghua University(清华大学求真书院)

Rui Lu, Runzhe Wang, Kaifeng Lyu, Xitai Jiang, Gao Huang, Mengdi Wang

AI总结:

本文研究扩散模型的文本幻觉问题,发现其源于去噪网络的局部生成偏置导致无法捕获全局结构,该现象跨架构存在,并提供了理论分析。

摘要:

基于分数的扩散模型在生成逼真图像、音频和视频数据方面取得了令人难以置信的性能。尽管这些模型能生成具有惊人细节的高质量样本,但它们经常引入不真实的伪影,例如扭曲的手指或毫无意义的幻觉文本。本文重点关注文本幻觉,即扩散模型能正确生成单个符号,却以毫无意义的方式将它们组合起来。通过实验探究,我们一致观察到这种现象归因于网络的局部生成偏置。去噪网络倾向于产生严重依赖高度相关的局部区域的输出,特别是当数据分布的不同维度几乎成对独立时。这种行为导致生成过程将全局分布分解为每个符号的独立分布,最终无法捕获包括底层语法在内的全局结构。有趣的是,这种偏置在各种去噪网络架构中持续存在,包括 MLP 和具有建模全局依赖结构的 transformers。这些发现还提供了见解,以理解由于去噪模型中的隐式偏置导致的超越文本的其他类型的幻觉。此外,我们理论上分析了涉及两层 MLP 在超立方体上学习奇偶点的特定情况的训练动态,为其潜在机制提供了解释。

英文摘要:

Score-based diffusion models have achieved incredible performance in generating realistic images, audio, and video data. While these models produce high-quality samples with impressive details, they often introduce unrealistic artifacts, such as distorted fingers or hallucinated texts with no meaning. This paper focuses on textual hallucinations, where diffusion models correctly generate individual symbols but assemble them in a nonsensical manner. Through experimental probing, we consistently observe that such phenomenon is attributed it to the network's local generation bias. Denoising networks tend to produce outputs that rely heavily on highly correlated local regions, particularly when different dimensions of the data distribution are nearly pairwise independent. This behavior leads to a generation process that decomposes the global distribution into separate, independent distributions for each symbol, ultimately failing to capture the global structure, including underlying grammar. Intriguingly, this bias persists across various denoising network architectures including MLP and transformers which have the structure to model global dependency. These findings also provide insights into understanding other types of hallucinations, extending beyond text, as a result of implicit biases in the denoising models. Additionally, we theoretically analyze the training dynamics for a specific case involving a two-layer MLP learning parity points on a hypercube, offering an explanation of its underlying mechanism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑