Long-context inputs in large language models (LLMs) often suffer from the "lost in the middle" problem, where critical information becomes diluted or ignored due to excessive length. Context compression methods aim to address this by reducing input size, but existing approaches struggle with balancing information preservation and compression efficiency. We propose Adaptive Task-Aware Compressor (ATACompressor), which dynamically adjusts compression based on the specific requirements of the task. ATACompressor employs a selective encoder that compresses only the task-relevant portions of long contexts, ensuring that essential information is preserved while reducing unnecessary content. Its adaptive allocation controller perceives the length of relevant content and adjusts the compression rate accordingly, optimizing resource utilization. We evaluate ATACompressor on three QA datasets: HotpotQA, MSMARCO, and SQUAD-showing that it outperforms existing methods in terms of both compression efficiency and task performance. Our approach provides a scalable solution for long-context processing in LLMs. Furthermore, we perform a range of ablation studies and analysis experiments to gain deeper insights into the key components of ATACompressor.
School of Mathematics, Northwest University(西北大学数学学院);Shanghai Institute for Mathematics and Interdisciplinary Sciences(上海数学与交叉科学研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Cheriton School of Computer Science, University of Waterloo(滑铁卢大学切尔顿计算机科学学院)
A Survey on Model Compression for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
University of Texas at Austin(德克萨斯大学奥斯汀分校);Drexel University(德雷塞尔大学);MIT(麻省理工学院);UIUC(伊利诺伊大学厄巴纳-香槟分校);Duke University(杜克大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);Center for AI Safety(人工智能安全中心);University of California, Berkeley(加州大学伯克利分校);University of Chicago(芝加哥大学)
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
Harbin Institute of Technology(哈尔滨工业大学);University of Science and Technology of China(中国科学技术大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Shandong Agricultural University(山东农业大学);Tencent Inc.(腾讯公司);Anhui University(安徽大学)
PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zhejiang University(浙江大学);CAS Key Laboratory of AI Security, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);Fudan University(复旦大学)
ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学信息中心);School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院)
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, AAIS, Peking University(北京大学人工智能研究院数据科学中心);Meituan(美团);Meta AI;National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);BIGAI, Beijing, China(北京通用人工智能研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);School of Computer Science and Cyber Engineering, Guangzhou University(广州大学计算机科学与网络工程学院)
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Ping An Technology (Shenzhen) Co., Ltd., China(平安科技(深圳)有限公司,中国);University of Science and Technology of China(中国科学技术大学);University of Electronic Science and Technology of China(电子科技大学)
Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression
Kahlert School of Computing, University of Utah(犹他大学卡勒特计算学院);Google DeepMind(谷歌DeepMind);Scientific Computing and Imaging Institute, University of Utah(犹他大学科学计算与成像研究所)
An Empirical Study on Prompt Compression for Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));South China University of Technology(华南理工大学);University of Science and Technology of China(中国科学技术大学)
The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));National University of Singapore(新加坡国立大学)
Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院,中国沈阳);NiuTrans Research, Shenyang, China(沈阳NiuTrans研究院,中国);Kunming University of Science and Technology, Kunming, China(昆明理工大学,中国昆明)
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models
Tsinghua University(清华大学);Shanghai University of Finance and Economics(上海财经大学);Institute for AI, Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Peking University(北京大学)