发表机构
Universität Trier（特里尔大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究将Transformer的KV缓存视为四阶张量，通过谱分析发现键和特征模式低秩而头和层模式满秩，比较四种张量分解后，Tucker在2-5倍压缩比下误差最低，且键和值的最佳表示不同。
AI 中文摘要
自回归Transformer的键值（KV）缓存可视为一个四阶张量，其维度涵盖注意力头、令牌、特征和分组层。我们在Mistral-7B-v0.3和LLaMA-2-13B上测量了所有四种模式展开的奇异值谱，并在匹配存储条件下比较了四种标准张量分解：Tucker、CP、张量列（tensor train）和t-SVD。谱将四个轴分为两类：令牌和特征模式具有低秩结构，尤其是键；头和层模式几乎满秩，在任何实际误差水平下都难以压缩。在四种分解中，Tucker在从2倍到5倍的每个压缩比下均实现最低的重建误差，因为它可以保持满秩模式不变。与二维展开基线的比较表明，键和值的最佳表示不同：二维方法在键上实现更低的误差，而四维Tucker在匹配存储下对值实现更低的误差。一个模式固定定理仅从测量的谱即可证明满秩保持。另外两个谱特性影响可压缩模式而不触及满秩模式：在每个压缩比下，值的误差下限高于键；并且后RoPE键在两种模型上损失了其前RoPE可压缩性的41%至64%。
英文摘要
The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Among the four decompositions, Tucker achieves the lowest reconstruction error at every compression ratio from $2\times$ to $5\times$, because it can leave the full-rank modes untouched. Comparisons with two-dimensional unfolding baselines show that the preferred representation differs between keys and values: 2D methods achieve lower key error, while four-way Tucker achieves lower value error at matched storage. A mode-pinning theorem certifies the full-rank preservation from the measured spectra alone. Two further spectral properties affect the compressible modes without touching the full-rank ones: values reach a higher error floor than keys at every ratio, and post-RoPE keys lose $41\%$ - $64\%$ of their pre-RoPE compressibility on both models.
Comments18 pages, 3 figures, 8 tables. Submitted to SIAM Journal on Matrix Analysis and Applications (SIMAX)
