并非所有思考都需要HBM:面向LLM推理的语义感知内存层次结构
Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning
AI总结:
本文提出一种语义感知内存层次结构,通过累积注意力评分将token分为四个层级,减少HBM占用并提升推理准确性,实验表明仅3%的淘汰率可保持91%的GSM8K准确率。
摘要:
推理LLM生成数千个思维链token,其KV缓存必须驻留于稀缺的GPU HBM。主流方法永久移除低重要性token,导致准确性降至0-2.5%。本文提出语义感知内存层次结构,将token分为HBM、DDR、压缩和淘汰四级,通过累积注意力评分进行排序。低重要性token被移动到CPU内存而非销毁,在每次注意力步骤前预加载回全精度,贡献与未离开GPU时相同。本文将其形式化为零近似误差卸载,并得出核心发现:准确性仅取决于永久淘汰的token数量(淘汰率),而非剩余HBM中的token数量。通过控制HBM和淘汰率的3x3网格验证,发现该结论在三种模型规模(7B-32B)和四个基准测试中均成立。仅3%的淘汰率可使GSM8K保持91%的全缓存准确性,MATH-500保持71%。在14B规模下,该结构与未压缩基线(90% vs. 86%)相当,同时将HBM占用减半。与当前最先进的R-KV方法在相同设置下的对比实验表明,其性能仅为0-32%。系统原型显示,这种保存的代价是适度的——5-7%的传输开销,并且生产批次大小下的规模分析预计可节省2-48 GB的HBM空间。
英文摘要:
Reasoning LLMs produce thousands of chain-of-thought tokens whose KV cache must reside in scarce GPU HBM. The dominant response -- permanently evicting low-importance tokens -- is catastrophic for reasoning: accuracy collapses to 0-2.5% when half the cache is removed. We ask a different question: must every token live in HBM, or can some live elsewhere? We introduce a semantics-aware memory hierarchy that sorts tokens into four tiers -- HBM, DDR, compressed, and evicted -- using cumulative attention scoring. Low-importance tokens are moved to CPU memory rather than destroyed; before each attention step they are prefetched back at full precision, contributing exactly the same terms as if they had never left the GPU. We formalize this as zero-approximation-error offloading and derive our central finding: accuracy depends solely on how many tokens are permanently discarded (the eviction ratio), not on how many remain in HBM. A controlled 3x3 grid over HBM and eviction ratios confirms this across three model scales (7B-32B) and four benchmarks. With only 3% eviction, the hierarchy retains 91% of full-cache accuracy on GSM8K and 71% on MATH-500 (n=200); at 14B scale it matches the uncompressed baseline (90% vs. 86%) while halving HBM occupancy. A head-to-head reproduction of R-KV -- the current SOTA eviction method -- on our setup achieves only 0-32% at comparable budgets. A system prototype with real GPU-CPU data movement shows that the price of this preservation is modest -- 5-7% transfer overhead -- and scaling analysis projects 2-48 GB HBM savings at production batch sizes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。