DoubleDipper：通过上下文复用改进长上下文大语言模型
DoubleDipper: Improving Long-Context LLMs via Context Recycling
AI总结：
针对长上下文大语言模型表现不佳的问题，提出DoubleDipper上下文学习方法，通过复用上下文生成少样本示例并增加相关段落识别步骤，在多模型长上下文问答任务上平均提升16个百分点，且能泛化至多跳任务。
摘要：
尽管大语言模型（LLMs）近年来取得了进展，但它们在涉及长上下文的任务上的表现仍不理想。在本研究中，我们提出了DoubleDipper，这是一种新颖的上下文学习（In-Context-Learning）方法，通过复用上下文为长上下文问答任务自动生成少样本示例。具体而言，给定一个长输入上下文（1-3k个token）和一个查询，我们从给定上下文中生成额外的查询-输出对作为少样本示例，同时仅引入一次上下文。这确保了示例演示利用与目标查询相同的上下文，同时仅向提示中添加少量token。我们进一步通过指示模型在给出答案前明确识别相关段落来增强每个演示，这在提升性能的同时为答案来源提供了细粒度的归因。我们在多个LLMs上应用了我们的方法，并在各种长上下文问答数据集上获得了显著提升（跨模型平均提升+16个绝对百分点）。令人惊讶的是，尽管仅引入了单跳上下文学习示例，LLMs通过我们的方法成功泛化到了多跳长上下文问答任务。
英文摘要：
Despite recent advancements in Large Language Models (LLMs), their performance on tasks involving long contexts remains sub-optimal. In this work, we propose DoubleDipper, a novel In-Context-Learning method that automatically generates few-shot examples for long context QA tasks by recycling contexts. Specifically, given a long input context (1-3k tokens) and a query, we generate additional query-output pairs from the given context as few-shot examples, while introducing the context only once. This ensures that the demonstrations are leveraging the same context as the target query while only adding a small number of tokens to the prompt. We further enhance each demonstration by instructing the model to explicitly identify the relevant paragraphs before the answer, which improves performance while providing fine-grained attribution to the answer source. We apply our method on multiple LLMs and obtain substantial improvements (+16 absolute points on average across models) on various QA datasets with long context. Surprisingly, despite introducing only single-hop ICL examples, LLMs successfully generalize to multi-hop long-context QA using our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
