Large language models (LLMs) have achieved huge success in numerous natural language process (NLP) tasks. However, it faces the challenge of significant resource consumption during inference. In this paper, we aim to improve the inference efficiency of LLMs by prompt caching, i.e., if the current prompt can be answered by the same response of a previous prompt, one can directly utilize that previous response without calling the LLM. Specifically, we focus on the prediction accuracy of prompt caching for single-round question-answering tasks via embedding similarity. The existing embeddings of prompts mostly focus on whether two prompts are semantically similar, which is not necessarily equivalent to whether the same response can answer them. Therefore, we propose a distillation-based method to fine-tune the existing embeddings for better caching prediction. Theoretically, we provide finite-sample guarantees for the convergence of our method under different types of loss functions. Empirically, we carefully construct a hard dataset based on Kwiatkowski et al. (2019) where the existing embedding model (Wang et al., 2022) only achieves an AUC of 0.51. We then fine-tune the above embedding model, which significantly improves the AUC of caching prediction from 0.51 to 0.81. We also conduct simulations demonstrating that our trained models achieve better caching efficiency than the previous embedding model.
Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
University of Washington（华盛顿大学）；Carnegie Mellon University（卡内基梅隆大学）；The Chinese University of Hong Kong（香港中文大学）；City University of Hong Kong（香港城市大学）；Microsoft Research（微软研究院）
S$^4$R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）
Are Optimal Algorithms Still Optimal? Rethinking Sorting in LLM-Based Pairwise Ranking with Batching and Caching
Departamento de Computación, FCEyN, Universidad de Buenos Aires（布宜诺斯艾利斯大学计算机系）；Instituto de Ciencias de la Computación, FCEyN, Universidad de Buenos Aires（布宜诺斯艾利斯大学计算机科学研究所）；Lumina Labs（Lumina实验室）
Consistent and Distinctive: LLM Benchmark Efficiency via Maximum Independent Set Prompt Selection on Similarity Graphs
Computer Systems Department, Jožef Stefan Institute, Ljubljana, Slovenia（计算机系统部，乔塞夫·斯塔芬研究所，卢布尔雅那，斯洛文尼亚）；Jožef Stefan International Postgraduate School, Ljubljana, Slovenia（乔塞夫·斯塔芬国际研究生学院，卢布尔雅那，斯洛文尼亚）；Center for Astrophysics and Cosmology, University of Nova Gorica, Nova Gorica, Slovenia（天体物理与宇宙学中心，诺瓦戈里察大学，诺瓦戈里察，斯洛文尼亚）
Gradient Co-occurrence Analysis for Detecting Unsafe Prompts in Large Language Models
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；IT Innovation and Research Center, Huawei Technologies（华为技术有限公司IT创新研究中心）；Artificial Intelligence Academy, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Expressive Prompting: Improving Emotion Intensity and Speaker Consistency in Zero-Shot TTS
Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部认知计算与应用天津市重点实验室）；Kuaishou Technology Co., Ltd（快手科技有限公司）；Institute of Artificial Intelligence, China Telecom（中国电信人工智能研究院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Huiyan Technology (Tianjin) Co., Ltd（慧眼科技（天津）有限公司）
Investigating the Influence of Prompt and Response Languages on LLM Content Generation
Université de Technologie de Compiègne（贡比涅技术大学）；Norwegian University of Science and Technology (NTNU)（挪威科技大学）；SimulaMet（西穆拉梅特研究院）；HUTECH University（胡志明市技术大学）
A Similarity Paradigm Through Textual Regularization Without Forgetting
Shanghai Jiao Tong University（上海交通大学）；Hong Kong Baptist University（香港浸会大学）；Northeastern University（东北大学）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Archer: Adaptive Reuse of Cached Hidden States for Efficient Rollback in Diffusion Language Models
School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；College of Artificial Intelligence, Nankai University（南开大学人工智能学院）；School of Computer Science, Peking University（北京大学计算机科学学院）
