arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07528cs.LG
收藏

QuickLLaMA:面向大型语言模型的查询感知推理加速

QuickLLaMA: Query-aware Inference Acceleration for Large Language Models

CUHK(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);SmartMore(思必驰)

Jingyao Li, Han Shi, Xin Jiang, Zhenguo Li, Hong Xu, Jiaya Jia

AI总结:

针对大型语言模型处理长上下文时难以捕捉长距离依赖的问题,提出了无需额外训练的查询感知推理系统Q-LLM,通过关注与查询相关的记忆数据在固定窗口内提取信息,并在多个长文本基准测试中显著超越当前SOTA。

摘要:

大型语言模型(LLMs)理解和推理长上下文的能力对于多个领域的进步至关重要。然而,它们在捕捉序列内的长距离依赖以深入理解语义方面仍然存在困难。为了解决这个问题,我们引入了面向LLMs的查询感知推理(Q-LLM),这是一种旨在像人类认知一样处理长序列的系统。通过关注与给定查询相关的记忆数据,Q-LLM能够在固定的窗口大小内准确捕捉相关信息,并为查询提供精确的答案。它不需要额外的训练,并且可以与任何LLMs无缝集成。使用LLaMA3的Q-LLM(QuickLLaMA)可以在30秒内阅读《哈利·波特》并准确回答问题。在广泛认可的基准测试中,Q-LLM在LLaMA3上较当前最优(SOTA)提升了7.17%,在Mistral上于$\infty$-bench提升了3.26%。在Needle-in-a-Haystack和BABILong任务中,Q-LLM较当前SOTA分别提升了7.0%和6.1%。我们的代码可在 https://github.com/dvlab-research/Q-LLM 找到。

英文摘要:

The capacity of Large Language Models (LLMs) to comprehend and reason over long contexts is pivotal for advancements in diverse fields. Yet, they still stuggle with capturing long-distance dependencies within sequences to deeply understand semantics. To address this issue, we introduce Query-aware Inference for LLMs (Q-LLM), a system designed to process extensive sequences akin to human cognition. By focusing on memory data relevant to a given query, Q-LLM can accurately capture pertinent information within a fixed window size and provide precise answers to queries. It doesn't require extra training and can be seamlessly integrated with any LLMs. Q-LLM using LLaMA3 (QuickLLaMA) can read Harry Potter within 30s and accurately answer the questions. On widely recognized benchmarks, Q-LLM improved by 7.17% compared to the current state-of-the-art on LLaMA3, and by 3.26% on Mistral on the $\infty$-bench. In the Needle-in-a-Haystack and BABILong task, Q-LLM improved upon the current SOTA by 7.0% and 6.1%. Our code can be found in https://github.com/dvlab-research/Q-LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑