arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16760cs.CRcs.DC
收藏

面向宽域网络的隐私保护分割推断与推测解码

Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks

Michael Cunningham

AI总结:

本文提出一种隐私保护的大语言模型分割推断系统,通过非对称层分割和lookahead解码技术,在宽域网络上实现高吞吐量和隐私安全的LLM推断。

摘要:

我们提出一个实用系统,用于隐私保护的大语言模型(LLM)推断,将transformer模型分割到受信任的本地GPU和不可信的云GPU之间,仅通过网络传输中间激活值。我们的系统解决了在高延迟宽域网络(WAN)上进行自回归LLM解码的独特挑战,贡献包括:(1)非对称层分割,其中嵌入和解嵌层保持本地,确保原始令牌从未离开受信任设备;(2)首次将lookahead解码应用于宽域网络上的分割推断,通过多个令牌每迭代次摊销网络往返延迟;(3)经验性反向攻击评估显示,分割深度提供了可调节的隐私-性能权衡——攻击者在2层分割时可恢复约59%的令牌,但在8层分割时仅能恢复约35%,且对吞吐量影响极小;(4)消融实验显示,n-gram推测在解码步骤中平均接受1.2-1.3个令牌(峰值为7个,见代码),接受率在不同模型规模上保持一致;(5)形式验证表明,lookahead解码在贪心argmax下会产生与顺序解码相同的令牌输出,无质量下降;(6)在Mistral NeMo 12B(40层)上的扩展验证,证明该系统仅需4.9 GB本地VRAM即可扩展到更大模型,并与7B吞吐量相匹配。在Mistral 7B和NeMo 12B上通过约80ms宽域网络链路评估,我们的系统在lookahead解码下分别达到7B的8.7-9.3 tok/s和12B的7.8-8.7 tok/s,具有验证误差小于6.2%的RTT分解模型,预测在20ms RTT下可达到15-19 tok/s。

英文摘要:

We present a practical system for privacy-aware large language model (LLM) inference that splits a transformer between a trusted local GPU and an untrusted cloud GPU, communicating only intermediate activations over the network. Our system addresses the unique challenges of autoregressive LLM decoding over high-latency wide-area networks (WANs), contributing: (1) an asymmetric layer split where embedding and unembedding layers remain local, ensuring raw tokens never leave the trusted device; (2) the first application of lookahead decoding to split inference over WANs, amortizing network round-trip latency across multiple tokens per iteration; (3) an empirical inversion attack evaluation showing that split depth provides a tunable privacy-performance tradeoff -- an attacker can recover ~59%% of tokens at a 2-layer split but only ~35%% at an 8-layer split, with minimal throughput impact; (4) ablation experiments showing that n-gram speculation accepts 1.2-1.3 tokens per decoding step on average (peak of 7 observed on code), with acceptance rates consistent across model scales; (5) formal verification that lookahead decoding produces token-identical output to sequential decoding under greedy argmax, with zero quality degradation; and (6) scaling validation on Mistral NeMo 12B (40 layers), demonstrating that the system generalizes to larger models with only 4.9 GB local VRAM and matching 7B throughput. Evaluated on Mistral 7B and NeMo 12B over a ~80ms WAN link, our system achieves 8.7-9.3 tok/s (7B) and 7.8-8.7 tok/s (12B) with lookahead decoding, with an RTT decomposition model (validated at <6.2%% cross-validation error) projecting 15-19 tok/s at 20ms RTT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑