Large language models (LLMs) are great at processing multiple natural language processing tasks, but their abilities are constrained by inferior performance with long context, slow inference speed, and the high cost of computing the results. Deploying LLMs with precise and informative context helps users process large-scale datasets more effectively and cost-efficiently. Existing works rely on compressing long prompt contexts into soft prompts. However, soft prompt compression encounters limitations in transferability across different LLMs, especially API-based LLMs. To this end, this work aims to compress lengthy prompts in the form of natural language with LLM transferability. This poses two challenges: (i) Natural Language (NL) prompts are incompatible with back-propagation, and (ii) NL prompts lack flexibility in imposing length constraints. In this work, we propose a Natural Language Prompt Encapsulation (Nano-Capsulator) framework compressing original prompts into NL formatted Capsule Prompt while maintaining the prompt utility and transferability. Specifically, to tackle the first challenge, the Nano-Capsulator is optimized by a reward function that interacts with the proposed semantics preserving loss. To address the second question, the Nano-Capsulator is optimized by a reward function featuring length constraints. Experimental results demonstrate that the Capsule Prompt can reduce 81.4% of the original length, decrease inference latency up to 4.5x, and save 80.1% of budget overheads while providing transferability across diverse LLMs and different datasets.
Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明的负责任人工智能与数据分析(PRADA)实验室);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);South China University of Technology(华南理工大学);University of Macau(澳门大学);Nankai University(南开大学);University of Copenhagen(哥本哈根大学);Stony Brook University(纽约州立大学石溪分校);SDAIA-KAUST AI(沙特数据与人工智能总局-阿卜杜拉国王科技大学人工智能机构);Ant Group(蚂蚁集团)
An Empirical Study on Prompt Compression for Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));South China University of Technology(华南理工大学);University of Science and Technology of China(中国科学技术大学)
PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Zhejiang University(浙江大学);CAS Key Laboratory of AI Security, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);Fudan University(复旦大学)
Dynamic Compressing Prompts for Efficient Inference of Large Language Models
School of Software Engineering, South China University of Technology(华南理工大学软件学院);Pazhou Lab(琶洲实验室);School of Future Technology, South China University of Technology(华南理工大学未来技术学院);Peng Cheng Laboratory(鹏城实验室);Hong Kong Polytechnic University(香港理工大学);Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications(重庆邮电大学计算机科学与技术学院)
Ministry of Education Key Laboratory of Bioinformatics, BNRIST Bioinformatics Division, Department of Automation, Tsinghua University(教育部生物信息学重点实验室、BNRIST生物信息学分部、自动化系、清华大学);School of Computing and Artificial Intelligence, Fuyao University of Science and Technology(计算与人工智能学院、福耀科技大学);Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系、南方科技大学)
PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));South China University of Technology(华南理工大学);Singapore Management University(新加坡管理大学)
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University(上海教育委员会智能交互与认知工程重点实验室);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海Web3可信数据流通与治理重点实验室);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);Xiaomi, Beijing, China(小米(北京))
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校)
Perception Compressor: A Training-Free Prompt Compression Framework in Long Context Scenarios
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Pengcheng Laboratory(鹏城实验室);Ant Group(蚂蚁集团);School of Artificial Intelligence, Sun Yat-sen University(中山大学人工智能学院)
Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devices
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州高等研究院)