从架构到输出：大语言模型中幻觉的结构性起源及数据的放大作用
From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
AI总结：
本文分析大语言模型幻觉的结构性根源，指出自注意力、最大似然估计训练目标和自回归解码三个架构决策构成复合失效系统，并揭示数据病理如何放大这些脆弱性。
摘要：
大语言模型会产生幻觉——生成流畅、自信但事实错误的输出——这种一致性跨越代际和规模。现有分类法按输出类型对幻觉进行分类，区分内在与外在失败以及忠实性与事实性偏差。这些框架在描述上严谨，但未能识别产生特定实例的内部机制。本文将幻觉分析为三个架构决策的结构性后果，这些决策共同构成一个复合失效系统。自注意力的共现学习用统计邻近性替代语义含义，导致实体混淆、事实错误归因和语义漂移。最大似然估计训练目标在无事实约束下优化下一个词元概率，奖励统计上合理的输出，无论其真值如何。自回归解码在暴露偏差下的永久从左到右承诺确保单个错误词元级联向前传递整个输出序列而无法修正。数据集病理——长尾缺陷、训练偏差和合成污染——放大了这些脆弱性，但并非独立导致它们。我们做出三项贡献。首先，我们将每个机制映射到Alansari和Luqman分类法中的特定输出类别，将内在幻觉定位于自注意力，外在幻觉定位于MLE，逻辑不一致定位于自回归解码。其次，我们表明每个常被引用的数据集病理利用这些机制之一，而非独立产生幻觉。第三，我们识别出仅基于输出类型分类的诊断局限性，并将其与推理层缓解方法进行对比。
英文摘要：
Large language models produce fluent, confident, factually wrong output. Existing taxonomies classify these failures by output type -- intrinsic versus extrinsic, faithfulness versus factuality -- but say nothing about which computational component produced a given failure. We ask what would be required to attribute an individual hallucination to a specific component of the decoder-only stack. We treat three components -- self-attention's associative retrieval, the maximum-likelihood pretraining objective, and autoregressive commitment under exposure bias -- as candidate failure surfaces, justify their separability rather than assuming it, and specify an attribution procedure requiring only sampling access: an ordered set of three interventions on prefix, context, and frequency competition, together with a validation design based on independent annotation and a classifier baseline. We state five falsifiable predictions and identify competing accounts each would discriminate against. We analyse how instruction tuning, RLHF, DPO, retrieval augmentation, scale, and calibration bear on the argument. We execute a direct, pre-registered test of the commitment prediction (P3) across three model families: substituting a correct continuation at the point of divergence reduces downstream failing claims by 46.7 percentage points relative to baseline (p<10^-9). However, a wrong-fact substitution reduces errors at a statistically indistinguishable rate, and the model answers correctly in isolation on only 2.2% of items where substitution succeeded -- a genuine partial result rather than a confirmation. Dataset pathologies amplify each component without originating failure independently, supporting an asymmetric-dependence claim: components are necessary intermediaries for data-induced failure, but data defects are not necessary for component-induced failure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
