DoLa：通过对比层解码提升大语言模型的事实性
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
AI总结：
针对大语言模型易产生幻觉的问题，提出无需外部知识和微调的DoLa对比层解码策略，通过对比前后层logits差异挖掘事实知识，在多类任务中显著提升模型真实性。
摘要：
尽管大语言模型（LLMs）能力出众，却容易产生幻觉，即生成的内容与预训练期间接触到的事实不符。我们提出了一种简单的解码策略，用于减少预训练大语言模型的幻觉，该策略无需依赖检索到的外部知识，也无需额外微调。我们的方法通过对比将后层与前层投影到词汇空间得到的logits差异，来获取下一词元的分布，这利用了大语言模型中的事实知识通常定位于特定Transformer层的已有结论。我们发现，这种对比层解码（DoLa）方法能够更好地挖掘事实知识，减少错误事实的生成。DoLa在多项选择题任务和开放式生成任务中均持续提升了真实性，例如将LLaMA系列模型在TruthfulQA上的性能绝对提升了12-17个百分点，证明了其在让大语言模型可靠生成真实事实方面的潜力。
英文摘要：
Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating its potential in making LLMs reliably generate truthful facts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
