arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.05004cs.CLcs.AIcs.IRcs.LG
收藏

记不住长文档中的细节?你需要一些R&R

Can't Remember Details in Long Documents? You Need Some R&R

Thomson Reuters(汤姆森路透)

Devanshu Agrawal, Shang Gao, Martin Gajek

AI总结:

针对长上下文大语言模型易遗漏长文档中间信息的问题,提出结合重复提示与上下文内检索的R&R方法,在长文档问答任务上平均提升准确率16个点,且相比短上下文分块法可减少模型调用并降低准确率损失。

摘要:

长上下文大语言模型(LLM)在长文档问答(QA)等任务上颇具潜力,但它们往往会遗漏上下文文档中间的重要信息(arXiv:2307.03172v3)。本文提出$\textit{R&R}$——两种基于提示的新方法的组合,分别名为$\textit{reprompting}$(重复提示)和$\textit{in-context retrieval}$(ICR,上下文内检索),以缓解文档问答中的这一问题。在重复提示方法中,我们在整个上下文文档中周期性地重复提示指令,提醒大语言模型其原始任务。在ICR方法中,我们并非指示大语言模型直接回答问题,而是指示其检索与给定问题最相关的前$k$个段落编号,随后将这些段落作为简化上下文用于第二个问答提示。我们使用GPT-4 Turbo和Claude-2.1在长度最高达80k token的文档上测试了R&R,观察到问答准确率平均提升16个点。进一步分析表明,R&R之所以能提升长文档问答性能,是因为它缩短了相关上下文与指令之间的距离。最后我们证明,与短上下文分块方法相比,R&R支持使用更大的分块,减少大语言模型调用次数和输出token数,同时将准确率降幅降至最低。

英文摘要:

Long-context large language models (LLMs) hold promise for tasks such as question-answering (QA) over long documents, but they tend to miss important information in the middle of context documents (arXiv:2307.03172v3). Here, we introduce $\textit{R&R}$ -- a combination of two novel prompt-based methods called $\textit{reprompting}$ and $\textit{in-context retrieval}$ (ICR) -- to alleviate this effect in document-based QA. In reprompting, we repeat the prompt instructions periodically throughout the context document to remind the LLM of its original task. In ICR, rather than instructing the LLM to answer the question directly, we instruct it to retrieve the top $k$ passage numbers most relevant to the given question, which are then used as an abbreviated context in a second QA prompt. We test R&R with GPT-4 Turbo and Claude-2.1 on documents up to 80k tokens in length and observe a 16-point boost in QA accuracy on average. Our further analysis suggests that R&R improves performance on long document-based QA because it reduces the distance between relevant context and the instructions. Finally, we show that compared to short-context chunkwise methods, R&R enables the use of larger chunks that cost fewer LLM calls and output tokens, while minimizing the drop in accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑