arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.00274cs.CLcs.AI
收藏

QUITO:通过查询引导的上下文压缩加速长上下文推理

QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression

Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Beijing University of Posts and Telecommunications(北京邮电大学)

Wenshan Wang, Yihang Wang, Yixing Fan, Huaming Liao, Jiafeng Guo

AI总结:

本文提出查询引导注意力压缩方法QUITO,利用问题对上下文的注意力分布过滤无用信息,在NaturalQuestions和ASQA上显著优于基线,有效加速大语言模型长上下文推理。

摘要:

上下文学习(ICL)能力是大语言模型(LLM)成功的基础。近年来,上下文压缩引起了越来越多的关注,因为它可以大幅降低大语言模型的推理复杂度和计算成本。在本文中,我们提出了一种新颖的查询引导注意力压缩(QUITO)方法,该方法利用问题对上下文的注意力来过滤无用信息。具体来说,我们使用一个触发标记来计算上下文响应问题时的注意力分布。基于该分布,我们提出了三种不同的过滤方法,以满足上下文长度的预算约束。我们使用两个广泛使用的数据集,即 NaturalQuestions 和 ASQA,对 QUITO 进行了评估。实验结果表明,QUITO 在各种数据集和下游大语言模型上显著优于已有的基线方法,凸显了其有效性。我们的代码可在 https://github.com/Wenshansilvia/attention_compressor 获取。

英文摘要:

In-context learning (ICL) capabilities are foundational to the success of large language models (LLMs). Recently, context compression has attracted growing interest since it can largely reduce reasoning complexities and computation costs of LLMs. In this paper, we introduce a novel Query-gUIded aTtention cOmpression (QUITO) method, which leverages attention of the question over the contexts to filter useless information. Specifically, we take a trigger token to calculate the attention distribution of the context in response to the question. Based on the distribution, we propose three different filtering methods to satisfy the budget constraints of the context length. We evaluate the QUITO using two widely-used datasets, namely, NaturalQuestions and ASQA. Experimental results demonstrate that QUITO significantly outperforms established baselines across various datasets and downstream LLMs, underscoring its effectiveness. Our code is available at https://github.com/Wenshansilvia/attention_compressor.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑