TRACE: 通过无损压缩和精度缩放解锁有效的CXL带宽
TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
AI总结:
TRACE通过无损压缩和精度缩放技术,有效提升CXL带宽利用率,减少内存占用并提高吞吐量。
摘要:
大语言模型推理日益受到内存带宽的限制,尤其是在长上下文情况下,随着KV缓存的增长,瓶颈问题更加严重。CXL内存增加了卸载权重和KV的能力,但其链路和设备侧DDR带宽远低于HBM,因此一旦流量转移到CXL层级,解码就会出现停滞。许多CXL控制器开始添加通用的无损压缩,但直接将通用编码器应用于标准的字优先LLM张量进行压缩效果不佳,尤其是针对以token优先的KV流。我们提出了TRACE(用于压缩和弹性的交通减少架构),它保持了未修改的CXL.mem接口,但改变了设备内部的表示形式。它以通道优先的方式存储张量,并在压缩前应用特定于KV的变换,将混合字段的字转换为低熵平面流,使通用编码器能够进行压缩。相同的基质还允许通过只读取所需的位平面实现精度比例的获取。在公共LLM上,TRACE无损地将BF16权重足迹减少了25.2%,将BF16 KV足迹减少了46.9%,每层KV比率峰值达到2.69倍。在基于跟踪的系统建模中,一旦KV溢出到CXL,GPT-OSS-120B-MXFP4在128000个token时的吞吐量从16.28提高到68.99 tok/s(4.24倍)。DRAMSim3显示,在平面对齐的获取下,DRAM访问能耗可降低高达40.3%。一个7nm的SystemVerilog实现能够维持256GB/s的设备带宽。与具有通用内联无损压缩的CXL控制器相比,TRACE仅增加7.2%的面积、4.7%的功率和6.0%的负载到使用延迟,在2GHz和0.7V下。
英文摘要:
LLM inference is increasingly limited by memory bandwidth, and the bottleneck worsens at long context as the KV cache grows. CXL memory adds capacity to offload weights and KV, but its link and device-side DDR bandwidth are far below HBM, so decoding stalls once traffic shifts to the CXL tier. Many CXL controllers are starting to add generic \emph{lossless} compression, yet applying commodity codecs directly to standard word-major LLM tensors is largely ineffective, especially for token-major KV streams. We propose TRACE (\textbf{T}raffic-\textbf{R}educed \textbf{A}rchitecture for \textbf{C}ompression and \textbf{E}lasticity), which preserves the unmodified CXL.mem interface but changes the device-internal representation. It stores tensors in a channel-major, disaggregated bit-plane layout, and applies a KV-specific transform before compression, converting mixed-field words into low-entropy plane streams that commodity codecs can compress. The same substrate enables precision-proportional fetch by reading only the required bit-planes. Across public LLMs, TRACE reduces BF16 weight footprint by 25.2\% and BF16 KV footprint by 46.9\% losslessly, with per-layer KV ratios peaking at 2.69$\times$. In trace-driven system modeling, once KV spills to CXL, GPT-OSS-120B-MXFP4 improves throughput at 128k tokens from 16.28 to 68.99 tok/s (4.24$\times$). DRAMSim3 shows up to 40.3\% lower DRAM access energy under plane-aligned fetch. A 7\,nm SystemVerilog implementation sustains 256\,GB/s device bandwidth. Relative to a CXL controller with generic inline lossless compression, TRACE only adds 7.2\% area, 4.7\% power, and 6.0\% load-to-use latency at 2\,GHz and 0.7\,V.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。