arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.14906cs.CLcs.IR
收藏

边缘书写:面向长上下文检索的更好推理模式

Writing in the Margins: Better Inference Pattern for Long Context Retrieval

Writer Inc(Writer公司)

Melisa Russak, Umar Jamil, Christopher Bryant, Kiran Kamble, Axel Magnuson, Mateusz Russak, Waseem AlShikh

AI总结:

本文提出WiM推理模式,通过键值缓存分块预填充进行分段推理并生成中间信息,以极低计算开销无需微调显著提升长上下文检索任务性能,推理准确率平均提升7.5%,聚合任务F1提升超30%,并支持交互式检索设计。

摘要:

在本文中,我们介绍了边缘书写(WiM),这是一种为大型语言模型设计的新型推理模式,旨在优化面向检索任务中对长输入序列的处理。该方法利用键值缓存的分块预填充来执行分段推理,从而能够高效处理广泛的上下文,并生成和分类指导模型执行特定任务的中间信息(“边缘”)。此方法在略微增加计算开销的同时,显著提升了现成模型的性能,且无需微调。具体而言,我们观察到WiM在推理技能(HotpotQA、MultiHop-RAG)上平均提高了7.5%的准确率,在聚合任务(CWE)上F1分数提升了30.0%以上。此外,我们展示了所提出的模式如何契合交互式检索设计,该设计为最终用户提供关于上下文处理进度的持续更新,并精准指出相关信息如何整合到最终响应中。我们在 https://github.com/writer/writing-in-the-margins 上发布了使用Hugging Face Transformers库的WiM实现。

英文摘要:

In this paper, we introduce Writing in the Margins (WiM), a new inference pattern for Large Language Models designed to optimize the handling of long input sequences in retrieval-oriented tasks. This approach leverages the chunked prefill of the key-value cache to perform segment-wise inference, which enables efficient processing of extensive contexts along with the generation and classification of intermediate information ("margins") that guide the model towards specific tasks. This method increases computational overhead marginally while significantly enhancing the performance of off-the-shelf models without the need for fine-tuning. Specifically, we observe that WiM provides an average enhancement of 7.5% in accuracy for reasoning skills (HotpotQA, MultiHop-RAG) and more than a 30.0% increase in the F1-score for aggregation tasks (CWE). Additionally, we show how the proposed pattern fits into an interactive retrieval design that provides end-users with ongoing updates about the progress of context processing, and pinpoints the integration of relevant information into the final response. We release our implementation of WiM using Hugging Face Transformers library at https://github.com/writer/writing-in-the-margins.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑