AbridgeKV:将长推理压缩为学习型暂停标记的KV缓存
AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens
AI总结:
AbridgeKV通过学习型暂停标记压缩长推理的KV缓存,在446道竞赛题上实现缓存缩小3.5倍、推理速度提升3.2倍且精度与全KV模型相当。
摘要:
长思维链解码受内存限制:生成标记的KV缓存随测试时计算量线性增长。驱逐缓存条目无法解决该问题,因为从未在移除条目的缓存上训练过的模型会生成更长的内容,通常达到长度限制。我们转而训练模型压缩自身输出:AbridgeKV将每段完整的生成文本块压缩为几个学习型暂停标记的KV条目,这些是带有训练嵌入的额外输入标记,不产生输出。暂停标记编码完文本块后,AbridgeKV会释放该块的原始缓存,后续标记会转而关注暂停标记的KV条目。三项创新设计使其可行:暂停标记的KV条目占据其替换块内的位置,缓存预算根据各层条目的测量值在层间分配,且压缩针对相对较大的块。在来自AIME、HMMT、AMC、BRUMO、CMIMC和SMT的446道竞赛题(生成长度为32K标记)上,保留四分之一的生成缓存与同等训练的全KV模型在准确率和输出长度上匹配;相同预算下最强的已发布驱逐器生成的内容长得多,而在相同数据上重新训练的最接近的训练型压缩器则损失10个百分点。在vLLM中部署时,每个序列的缓存缩小3.5倍,饱和H100完成一道题的速度比使用全KV时快3.2倍。
英文摘要:
Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time compute being spent. Evicting cache entries does not fix this, because a model never trained on a cache with entries removed generates longer, often up to the length limit. We instead train the model to compress its own output. AbridgeKV compresses each completed block of generated text into the KV entries of a few learned pause tokens. These are extra input tokens with trained embeddings that produce no output. Once the pause tokens have encoded the block, AbridgeKV frees the block's original cache; subsequent tokens attend to the pause tokens' KV entries in its place. Three novel choices make this work: the pause tokens' KV entries take positions inside the block they replace, the cache budget is allocated across layers according to the measured value of each layer's entries, and compression operates on relatively large blocks. On 446 competition problems from AIME, HMMT, AMC, BRUMO, CMIMC and SMT with 32K-token generations, keeping a quarter of the generated cache matches an identically trained full-KV model in accuracy and output length; the strongest published evictor at a matched budget writes far longer, and the closest trained compressor, retrained on the same data, loses 10 points. Served in vLLM, the cache is 3.5x smaller per sequence and a saturated H100 finishes a problem 3.2x faster than with full KV.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。