Cache-to-Cache: Direct Semantic Communication Between Large Language Models
Tsinghua University(清华大学);Infinigence AI;The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学);SLAI;Shanghai AI Laboratory(上海人工智能实验室)
Multi-LLM systems harness the complementary strengths of diverse Large Language Models, achieving performance and efficiency gains that are not attainable by a single model. In existing designs, LLMs communicate through text, forcing internal representations to be transformed into output token sequences. This process both loses rich semantic information and incurs token-by-token generation latency. Motivated by these limitations, we ask: Can LLMs communicate beyond text? Oracle experiments show that enriching the KV-Cache semantics can improve response quality without increasing cache size, supporting KV-Cache as an effective medium for inter-model communication. Thus, we propose Cache-to-Cache (C2C), a new paradigm for direct semantic communication between LLMs. C2C uses a neural network to project and fuse the source model's KV-cache with that of the target model to enable direct semantic transfer. A learnable gating mechanism selects the target layers that benefit from cache communication. Compared with text communication, C2C utilizes the deep, specialized semantics from both models, while avoiding explicit intermediate text generation. Experiments show that C2C achieves 6.4-14.2% higher average accuracy than individual models. It further outperforms the text communication paradigm by approximately 3.1-5.4%, while delivering an average 2.5x speedup in latency. Our code is available at https://github.com/thu-nics/C2C.
SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size
School of Computer Science and Technology, University of Science and Technology of China(计算机科学与技术学院,科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州先进研究院,科学技术大学)
SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models
School of Computing Science, Simon Fraser University(西蒙菲莎大学计算科学学院);University of Mississippi(密西西比大学);Jiangxing Intelligence Inc.(江兴智联公司);The University of Hong Kong(香港大学)
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));CSE, The Hong Kong University of Science and Technology(香港科学与技术大学计算机科学与工程系);Guangzhou HKUST Fok Ying Tung Research Institute(广州HKUST福 Ying Tung研究 institute);Terminus Technologies
Transactional Attention: Semantic Sponsorship for KV-Cache Retention
National Institute of Electronics and Information Technology (NIELIT)(国家电子与信息技术研究院);Indian Institute of Information Technology, Allahabad (IIITA)(阿拉哈巴德印度信息技术学院)