Large language models (LLMs) enable state-of-the-art semantic capabilities to be added to software systems such as semantic search of unstructured documents and text generation. However, these models are computationally expensive. At scale, the cost of serving thousands of users increases massively affecting also user experience. To address this problem, semantic caches are used to check for answers to similar queries (that may have been phrased differently) without hitting the LLM service. Due to the nature of these semantic cache techniques that rely on query embeddings, there is a high chance of errors impacting user confidence in the system. Adopting semantic cache techniques usually requires testing the effectiveness of a semantic cache (accurate cache hits and misses) which requires a labelled test set of similar queries and responses which is often unavailable. In this paper, we present VaryGen, an approach for using LLMs for test input generation that produces similar questions from unstructured text documents. Our novel approach uses the reasoning capabilities of LLMs to 1) adapt queries to the domain, 2) synthesise subtle variations to queries, and 3) evaluate the synthesised test dataset. We evaluated our approach in the domain of a student question and answer system by qualitatively analysing 100 generated queries and result pairs, and conducting an empirical case study with an open source semantic cache. Our results show that query pairs satisfy human expectations of similarity and our generated data demonstrates failure cases of a semantic cache. Additionally, we also evaluate our approach on Qasper dataset. This work is an important first step into test input generation for semantic applications and presents considerations for practitioners when calibrating a semantic cache.
ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学信息中心）；School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）
VIS Lab, University of Amsterdam（阿姆斯特丹大学VIS实验室）；FunAI Lab, University of Technology Nuremberg（纽伦堡技术大学FunAI实验室）；CSAIL, Massachusetts Institute of Technology（麻省理工学院计算机科学与人工智能实验室）
SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models
School of Computing Science, Simon Fraser University（西蒙菲莎大学计算科学学院）；University of Mississippi（密西西比大学）；Jiangxing Intelligence Inc.（江兴智联公司）；The University of Hong Kong（香港大学）
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Alibaba Cloud Computing（阿里云计算）；Xiaohongshu Inc.（小红书公司）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
Nanjing University of Science and Technology（南京理工大学）；Alibaba Group（阿里巴巴集团）；Huzhou Normal University（湖州师范学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
KV Packet: Recomputation-Free Context-Independent KV Caching for LLMs
Technical University of Munich（慕尼黑技术大学）；Technical University of Darmstadt（达姆施塔特技术大学）；Zhejiang University（浙江大学）；Technische Universität Ilmenau（伊尔梅瑙技术大学）
Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle
School of Computer Science and Technology, University of Science and Technology of China（计算机科学与技术学院，中国科学技术大学）；University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究所，合肥综合性国家科学中心）；Suzhou Institute for Advanced Research, University of Science and Technology of China（苏州先进研究院，中国科学技术大学）
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
University of Wisconsin - Madison（威斯康星大学麦迪逊分校）；Peking University（北京大学）；Nanjing University（南京大学）；University of Surrey（Surrey 大学）；Qwen（通义实验室）；University of California - Riverside（加州大学河滨分校）；Microsoft（微软公司）
LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs
Institute of Artificial Intelligence, China Telecom (TeleAI)（中国电信人工智能研究院（TeleAI））；Shanghai Jiao Tong University（上海交通大学）；University at Buffalo（纽约州立大学布法罗分校）
A Survey on Large Language Model Acceleration based on KV Cache Management
Department of Computing, The Hong Kong Polytechnic University（计算系，香港理工大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Department of Computer Science and Technology（计算机科学与技术系）；Department of Computing and Data Science（计算与数据科学系）
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）
