Long-context language modeling is increasingly constrained by the Key-Value (KV) cache, whose memory and decode-time access costs scale linearly with the prefix length. This bottleneck has motivated a range of context-compression methods, from token-level summarization to recent optimization-based KV compression methods. These post-hoc methods operate on the KV cache of a fixed pretrained model, so their effectiveness is fundamentally limited by how well the model's internal representations can be compressed. In this work, we formalize the notion of KV compressibility and show that it is a property of the learned representations, rather than of the context alone. We prove that almost any sequence-to-vector function admits both highly compressible and inherently non-compressible transformer implementations, highlighting the need to guide transformers toward compressible representations during training. Motivated by this, we propose KV-Compression Aware Training (KV-CAT), a continued pretraining procedure that incentivizes the emergence of compressible representations. We introduce a train-time KV sparsification policy that masks KV slots during training. This forces the model to use fewer KV slots and encourages it to learn representations amenable to post-hoc compression. Empirically, we show that KV-CAT improves the quality-budget tradeoff of downstream compression methods across retrieval, long-context question answering, and perplexity-based evaluation of compressed-prefix continuation.
From TLinFormer to TConstFormer: The Leap to Constant-Time Transformer Attention: Achieving O(1) Computation and O(1) KV Cache during Autoregressive Inference
A Theory for Compressibility of Graph Transformers for Transductive Learning
University of British Columbia（不列颠哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）；Meta；CMU（卡内基梅隆大学）；Google Research（谷歌研究院）；UBC（不列颠哥伦比亚大学）；Amii（阿尔伯塔创新人工智能研究所）
Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
Center for Applied Mathematics（应用数学中心）；Cornell University（康奈尔大学）；Amazon Web Services（亚马逊网络服务）；Amazon Selling Partner Services（亚马逊销售合作伙伴服务）；Amazon Supply Chain Optimization Technologies（亚马逊供应链优化技术）
Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decoding
The University of Chicago（芝加哥大学）；Stevens Institute of Technology（史蒂文斯理工学院）；The University of Hong Kong（香港大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；The Simons Institute for the Theory of Computing at UC Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）
Tensor Cache: Eviction-conditioned Associative Memory for Transformers
Massachusetts Institute of Technology, Cambridge, MA, USA（麻省理工学院）；IBM Research, Cambridge, MA, USA（IBM研究院）；University of Toronto, Toronto, Canada（多伦多大学）
End-to-End Transformer Acceleration Through Processing-in-Memory Architectures
Department of Electrical and Computer Engineering, University of Virginia（弗吉尼亚大学电气与计算机工程系）；Department of Electrical and Computer Engineering, Duke University（杜克大学电气与计算机工程系）
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Microsoft（微软公司）；Rutgers University（罗格斯大学）；California Institute of Technology（加州理工学院）；Carnegie Mellon University（卡内基梅隆大学）
LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocation
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；Stepfun
