通过对比解码增强大语言模型的上下文理解能力
Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding
AI总结：
本文提出一种在推理时结合对比解码与对抗性不相关段落负样本的方法，以增强大语言模型在开放域问答中的上下文锚定能力，实验证明其优于现有方法且无需额外训练。
摘要：
大语言模型（LLMs）在文本生成过程中往往不能充分整合输入上下文，过度依赖模型参数中编码的先验知识，可能导致生成文本出现事实不一致或上下文不忠实的内容。LLMs 使用两种主要知识来源：1）来自预训练的先验（参数化）知识，以及 2）来自输入提示的上下文（非参数化）知识。本研究探讨了 LLMs 在生成过程中如何有效平衡这些知识来源这一开放问题，特别是在开放域问答的背景下。为了解决这一问题，我们提出了一种新方法，将对比解码与对抗性不相关段落作为负样本相结合，以增强生成过程中的鲁棒上下文锚定。值得注意的是，我们的方法在推理时运行，无需进一步训练。我们进行了全面的实验以证明其适用性和有效性，并提供了实证证据，展示其优于现有方法。我们的代码公开在：https://github.com/amazon-science/ContextualUnderstanding-ContrastiveDecoding。
英文摘要：
Large language models (LLMs) tend to inadequately integrate input context during text generation, relying excessively on encoded prior knowledge in model parameters, potentially resulting in generated text with factual inconsistencies or contextually unfaithful content. LLMs utilize two primary knowledge sources: 1) prior (parametric) knowledge from pretraining, and 2) contextual (non-parametric) knowledge from input prompts. The study addresses the open question of how LLMs effectively balance these knowledge sources during the generation process, specifically in the context of open-domain question answering. To address this issue, we introduce a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples to enhance robust context grounding during generation. Notably, our method operates at inference time without requiring further training. We conduct comprehensive experiments to demonstrate its applicability and effectiveness, providing empirical evidence showcasing its superiority over existing methodologies. Our code is publicly available at: https://github.com/amazon-science/ContextualUnderstanding-ContrastiveDecoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
