arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.02376cs.CL
收藏

在高压缩率下保留关键信息:面向 LLM 的 Query-Guided Compressor

Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Informatics, Xiamen University(厦门大学信息学院);ByteDance(字节跳动)

Zhiwei Cao, Qian Cao, Yu Lu, Ningxin Peng, Luyang Huang, Shanbo Cheng, Jinsong Su

AI总结:

针对 LLM 上下文压缩在高压缩率下丢失关键信息的问题,提出利用查询引导并结合动态压缩策略的 QGC,在多个问答数据集上实现稳定高性能并降低推理成本。

摘要:

大型语言模型的日益普及激发了人们对大型语言模型(LLM)上下文压缩的兴趣。然而,随着压缩率提高,以往方法的性能会急剧下降,有时甚至跌至闭卷水平。这种下降可归因于压缩过程中关键信息的丢失。我们的初步研究支持这一假设,并强调了在高压缩率下保留关键信息以维持模型性能的重要性。因此,我们提出 Query-Guided Compressor(QGC,查询引导压缩器),它利用查询来引导上下文压缩过程,从而在压缩后的上下文中有效保留关键信息。此外,我们采用了一种动态压缩策略。我们在问答任务上验证了所提出 QGC 的有效性,包括 NaturalQuestions、TriviaQA 和 HotpotQA 数据集。实验结果表明,即使在高压缩率下,QGC 也能持续表现良好,这同时在推理成本和吞吐量方面带来了显著收益。

英文摘要:

The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closed-book level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑