Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 70% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge transfer among requests. However, naively caching and reusing past responses leads to a big quality drop. In this paper, we introduce IC-Cache, a caching system that enables live LLM capability augmentation to improve serving efficiency: by leveraging historical request-response pairs from larger models as in-context examples, IC-Cache empowers small LLMs to imitate and even exceed the compositional abilities (e.g., reasoning) of their larger counterparts, enabling selective offloading of requests to reduce cost and latency. Achieving this live augmentation at scale introduces intricate trade-offs between response quality, latency, and system throughput. For a new request, IC-Cache efficiently selects similar, high-utility examples to prepend them to the new request's input. At scale, it adaptively routes requests across LLMs of varying capabilities, accounting for response quality and serving loads. IC-Cache employs a cost-aware cache replay mechanism that refines example quality offline to maximize online cache utility and efficiency. Evaluations on millions of realistic requests demonstrate that IC-Cache improves LLM serving throughput by 1.4-5.9x and reduces latency by 28-71% without hurting response quality.
PA-Cache: Evolving Learning-Based Popularity-Aware Content Caching in Edge Networks
Chongqing University(重庆大学);Binghamton University, State University of New York(纽约州立大学宾汉姆顿大学);Swinburne University of Technology(斯威本科技大学);Harbin Institute of Technology(哈尔滨工业大学)
随着无处不在和个性化服务的蓬勃发展,大量移动设备在网络上产生了越来越多的流量。因此,内容缓存逐渐延伸到网络边缘,以提供低延迟服务,提高服务质量,并减少冗余数据流量。与传统的内容分发网络相比,边缘网络中具有较小尺寸的高速缓存通常必须容纳更多突发请求。在本文中,我们提出了一个不断发展的学习为基础的内容缓存策略,命名为PA缓存在边缘网络。它自适应地学习随时间变化的内容流行度,并确定当缓存已满时应该替换哪些内容。与传统的深度神经网络(DNN)不同,DNN使用具有高计算复杂性的整个训练数据集来学习微调但可能过时或有偏见的预测模型,PA-Cache对大量内容特征集进行加权,并在随着时间的推移有更多请求到达时从浅到深地训练多层递归神经网络。我们广泛评估我们提出的PA缓存的性能从一个大型的在线视频点播服务提供商在现实世界中的痕迹。\rb{The results show that PA-Cache outperforms existing popular caching algorithms and approximates the optimal algorithm with only a 3.8\% performance gap when the cache percentage is 1.0\%}。与传统的基于DNN的方法相比,PA-Cache还显着降低了计算成本。
MVR-cache: Optimizing Semantic Caching via Multi-Vector Retrieval and Learned Prompt Segmentation
School of Computer Science, Peking University, Beijing, China(北京大学计算机科学学院,北京,中国);School of Information, Renmin University of China, Beijing, China(中国人民大学信息学院,北京,中国)
TDC-Cache: A Trustworthy Decentralized Cooperative Caching Framework for Web3.0
School of Electronic and Optical Engineering, Nanjing University of Science and Technology(南京理工大学电子与光学工程学院);College of Electronics and Information Engineering, Shenzhen University(深圳大学电子与信息工程学院);National Mobile Communications Research Laboratory, Southeast University(东南大学国家移动通信研究中心);Purple Mountain Laboratories(紫金山实验室);School of Electrical Engineering and Telecommunications, The University of New South Wales(新南威尔士大学电子与电信工程学院)
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Department of Computer Science, Johns Hopkins University, Baltimore, USA(约翰霍普金斯大学计算机科学系);Argonne National Laboratory, Lemont, USA(阿贡国家实验室);Parasail, Inc., San Mateo, USA(Parasail公司)