Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50\% reduction in context cost, resulting in a 36\% reduction in inference memory usage and a 32\% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.
Dynamic Compressing Prompts for Efficient Inference of Large Language Models
School of Software Engineering, South China University of Technology（华南理工大学软件学院）；Pazhou Lab（琶洲实验室）；School of Future Technology, South China University of Technology（华南理工大学未来技术学院）；Peng Cheng Laboratory（鹏城实验室）；Hong Kong Polytechnic University（香港理工大学）；Department of Computer Science and Technology, Chongqing University of Posts and Telecommunications（重庆邮电大学计算机科学与技术学院）
Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller
Shenzhen University（深圳大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；California Institute of Technology（加州理工学院）
Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations
Business Faculty Ansbach University of Applied Sciences（安斯巴赫应用技术大学商学院）；Faculty of Business University of Applied Science Ansbach（安斯巴赫应用技术大学商学院）；Faculty of Business University of Applied Science Heilbronn（海尔布隆应用技术大学商学院）
TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；MiLM Plus, Xiaomi Inc（小米公司MiLM Plus）
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
ETH Zurich（苏黎世联邦理工学院）；Allen Institute for AI（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
