Transformer models have driven breakthroughs across various language tasks by their strong capability to learn rich contextual representations. Scaling them to improve representation, however, often demands substantial memory and compute costs, such as the Key-Value (KV) cache used during auto-regressive decoding. Skip connections offer a promising way to improve representation without bloating resource usage, yet most prior works either improve expressivity while leaving KV costs unchanged, or reduce memory at the cost of weaker representation. In this work, we propose SkipV1Former, a Transformer variant that uses skip connections from the first layer's Value heads to strengthen model representation and reduce KV cache. Specifically, from the second block onward, each layer reuses half of its Value heads from the very first layer, while computing the other half as usual-cutting Value projections and V cache by nearly 50 \%. Theoretically, we show that routing uncompressed first-layer Values into deeper layers restores information lost to compression and accelerates the model's implicit mesa-optimization-a key pattern of Transformer in auto-regressive tasks. Empirically, across different model scales, SkipV1Former delivers consistent reductions of approximately 25 \% in KV cache while improving perplexity relative to standard Multi-Head Attention (MHA) Transformers and some advanced variants. Moreover, we propose a recipe for uptraining existing MHA Transformer checkpoints to SkipV1Former with only 10-15\% additional compute. Finally, SkipV1Former can seamlessly combine advanced methods like Group-Query Attention and Multi-Latent Attention to achieve further KV cache savings and performance improvement. When combined with YOCO, it cuts KV cache size by nearly 50 \% while still improving performance.
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Microsoft(微软公司);Rutgers University(罗格斯大学);California Institute of Technology(加州理工学院);Carnegie Mellon University(卡内基梅隆大学)
ChampionKV: Head-Wise Adaptive Budget Allocation for KV Cache Compression
Sun Yat-sen University(中山大学);The Chinese University of Hong Kong(香港中文大学);University of Electronic Science and Technology of China(电子科技大学);Tongji University(同济大学);Hong Kong University(香港大学)
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);School of Artificial Intelligence, Nanjing University(人工智能学院,南京大学);Stepfun
The Pennsylvania State University(宾夕法尼亚州立大学);University of Connecticut(康涅狄格大学);Carnegie Mellon University(卡内基梅隆大学);University of California, Los Angeles(加州大学洛杉矶分校)
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
University of Science and Technology of China(中国科学技术大学);State Key Lab of Processors, Institute of Computing Technology, CAS, Beijing, China(中国科学院计算技术研究所过程器重点实验室,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国)
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
Institute for Artificial Intelligence, Peking University(北京大学人工智能研究所);State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Tencent Youtu Lab, Shanghai, China(腾讯云智实验室,上海,中国)