通过解码器层跳跃减轻大型语言模型中的幻觉
Mitigating Hallucinations in Large Language Models Via Decoder Layer Skipping
AI总结：
本文提出DeLask框架，通过动态跳过易产生幻觉的解码器层，利用梯度下降的等价性检测并抑制错误信号，从而减轻LLM幻觉并提升可靠性。
摘要：
大型语言模型（LLM）在各种自然语言任务中表现出色，但其输出常常出现幻觉——与事实信息不符的内容。在这项工作中，我们对解码过程进行了全面的逐层分析，并揭示幻觉往往源自更深的解码器层。为了解决这个问题，我们引入了 extbf{DeLask}（ extbf{De}coder extbf{La}yer extbf{Sk}ipping），一种新颖的解码框架，它动态跳过容易产生幻觉的层。DeLask利用理论洞察，即$L$层Transformer的前向计算在条件上等价于$L$步梯度下降。我们通过计算连续解码步骤导出的梯度之间的余弦相似度来定义\emph{漂移值}，从而在下降方向反转时识别问题层。DeLask并非完全丢弃这些层，而是将其隐藏状态与前面层部分聚合，从而在抑制错误信号的同时保持一致性。跨不同LLM和基准的广泛实验表明，DeLask持续减轻幻觉并增强整体可靠性，为提升大规模语言模型的鲁棒性提供了一个轻量级且可泛化的解码框架。
英文摘要：
Large Language Models (LLMs) have achieved strong performance across diverse natural language tasks, yet their outputs often suffer from hallucinations -- content that is misaligned with factual information. In this work, we conduct a comprehensive layer-wise analysis of the decoding process and reveal that hallucinations tend to originate from deeper decoder layers. To address this issue, we introduce \textbf{DeLask} (\textbf{De}coder \textbf{La}yer \textbf{Sk}ipping), a novel decoding framework that dynamically skips layers prone to producing hallucinations. DeLask leverages the theoretical insight that the forward computation of an $L$-layer Transformer is conditionally equivalent to $L$ steps of gradient descent. We define a \emph{driftance value} by computing the cosine similarity between gradients derived from consecutive decoder steps, identifying problematic layers when the descent direction reverses. Rather than discarding such layers entirely, DeLask partially aggregates their hidden states with preceding layers, thereby preserving consistency while suppressing erroneous signals. Extensive experiments across diverse LLMs and benchmarks demonstrate that DeLask consistently mitigates hallucinations and enhances overall reliability, providing a lightweight and generalizable decoding framework for improving the robustness of large-scale language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
