The increasing prevalence of large language models (LLMs) such as GPT-4 in various applications has led to a surge in the size of prompts required for optimal performance, leading to challenges in computational efficiency. Prompt compression aims to reduce the inference cost by minimizing input tokens without compromising on the task performance. However, existing prompt compression techniques either rely on sub-optimal metrics such as information entropy or model it as a task-agnostic token classification problem that fails to capture task-specific information. To address these issues, we propose a novel and efficient reinforcement learning (RL) based task-aware prompt compression method. To ensure low latency requirements, we leverage existing Transformer encoder-based token classification model while guiding the learning process with task-specific reward signals using lightweight REINFORCE algorithm. We evaluate the performance of our method on three diverse and challenging tasks including text summarization, question answering and code summarization. We demonstrate that our RL-guided compression method improves the task performance by 8% - 189% across these three scenarios over state-of-the-art compression techniques while satisfying the same compression rate and latency requirements.
Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明的负责任人工智能与数据分析(PRADA)实验室);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);South China University of Technology(华南理工大学);University of Macau(澳门大学);Nankai University(南开大学);University of Copenhagen(哥本哈根大学);Stony Brook University(纽约州立大学石溪分校);SDAIA-KAUST AI(沙特数据与人工智能总局-阿卜杜拉国王科技大学人工智能机构);Ant Group(蚂蚁集团)
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University(上海教育委员会智能交互与认知工程重点实验室);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海Web3可信数据流通与治理重点实验室);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);Xiaomi, Beijing, China(小米(北京))
PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));South China University of Technology(华南理工大学);Singapore Management University(新加坡管理大学)
An Empirical Study on Prompt Compression for Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));South China University of Technology(华南理工大学);University of Science and Technology of China(中国科学技术大学)
PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zhejiang University(浙江大学);CAS Key Laboratory of AI Security, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);Fudan University(复旦大学)
Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Pengcheng Laboratory(鹏城实验室);Ant Group(蚂蚁集团);School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院)
Ministry of Education Key Laboratory of Bioinformatics, BNRIST Bioinformatics Division, Department of Automation, Tsinghua University(教育部生物信息学重点实验室、BNRIST生物信息学分部、自动化系、清华大学);School of Computing and Artificial Intelligence, Fuyao University of Science and Technology(计算与人工智能学院、福耀科技大学);Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系、南方科技大学)
Lexical Prompt Compression for Large Language Models: A Training-Free, Deterministic Pipeline with Empirical Pareto Analysis Across Eleven Task Categories
Dynamic Compressing Prompts for Efficient Inference of Large Language Models
School of Software Engineering, South China University of Technology(华南理工大学软件学院);Pazhou Lab(琶洲实验室);School of Future Technology, South China University of Technology(华南理工大学未来技术学院);Peng Cheng Laboratory(鹏城实验室);Hong Kong Polytechnic University(香港理工大学);Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications(重庆邮电大学计算机科学与技术学院)
BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection
School of Computer Technology and Application, Qinghai University(青海大学计算机技术与应用学院);Tsinghua University(清华大学);Ant Group Security and Intelligence Laboratory (SIL)(蚂蚁集团安全与智能实验室)
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);National University of Singapore(新加坡国立大学);Nanjing University of Posts and Telecommunications(南京邮电大学)
DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoning
Institute of Cyberspace Security, Zhejiang University of Technology A STAR Amazon Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology