The growing popularity of Large Language Models has sparked interest in context compression for Large Language Models (LLMs). However, the performance of previous methods degrades dramatically as compression ratios increase, sometimes even falling to the closed-book level. This decline can be attributed to the loss of key information during the compression process. Our preliminary study supports this hypothesis, emphasizing the significance of retaining key information to maintain model performance under high compression ratios. As a result, we introduce Query-Guided Compressor (QGC), which leverages queries to guide the context compression process, effectively preserving key information within the compressed context. Additionally, we employ a dynamic compression strategy. We validate the effectiveness of our proposed QGC on the Question Answering task, including NaturalQuestions, TriviaQA, and HotpotQA datasets. Experimental results show that QGC can consistently perform well even at high compression ratios, which also offers significant benefits in terms of inference cost and throughput.
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Alibaba Group(阿里巴巴集团)
AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算先进创新中心);School of Artificial Intelligence, Beihang University(北航人工智能学院);Beijing Academy of Blockchain and Edge Computing(北京区块链与边缘计算研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Stacked from One: Multi-Scale Self-Injection for Context Window Extension
Singapore University of Technology and Design (SUTD)(新加坡科技设计大学);Singapore Management University (SMU)(新加坡管理大学);Nanyang Technological University (NTU)(南洋理工大学);National University of Singapore (NUS)(新加坡国立大学)
From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition
DeepLang AI;Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Jiaotong University(北京交通大学)
SEER: Long-Context Reasoning via Selective Visual-Text Compression
The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);University of Cambridge(剑桥大学);University of Technology Sydney(悉尼科技大学);The University of North Carolina at Charlotte(北卡罗来纳大学夏洛特分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校)