一种面向共享内存网络的开关中心网络架构，用于加速大语言模型推理
A Switch-Centric In-Network Architecture for Accelerating LLM Inference in Shared-Memory Network
AI总结：
本文提出SCIN架构，通过在开关内直接访问附加加速器的内存区域，实现低延迟高带宽的All-Reduce，同时支持8位精度的In-Network Quantization，提升LLaMA-2模型推理效率。
摘要：
在网计算技术，如NVLink SHARP（NVLS），通过将All-Reduce等集体操作卸载到交换机，提供了一种解决LLM推理通信瓶颈的有希望的方法。然而，NVLS的加速器中心架构存在两个根本性限制：1）它依赖于GPU负载指令触发交换机内的减少操作，这意味着交换机减少的数据必须返回到启动GPU而不是直接广播，从而引入不必要的通信开销；2）由于其架构限制，NVLS无法卸载不能分解为内存语义指令的操作，如本文提出的在网量化（INQ）。因此，NVLS在推理期间的All-Reduce仍以16位精度运行，导致大量带宽浪费。为了解决这些限制，我们提出了SCIN，这是首个面向多加速器共享内存网络的开关中心在网架构，实现了低延迟和高带宽的All-Reduce。具体来说，我们引入了一个在交换机内的加速器（ISA），能够直接访问附加加速器的内存区域以进行在网处理，以及一个共同设计的通信织物，使这种访问具有极小的协议开销。SCIN通过消除冗余数据移动，比NVLS的All-Reduce延迟更低。此外，SCIN为All-Reduce启用INQ，将其精度降至8位，并几乎翻倍带宽，同时精度损失极小。我们还展示了SCIN的多FPGA原型以验证其可行性和有效性。对8GPU系统的模拟结果表明，我们的设计在小消息情况下将All-Reduce加速了高达8.7倍，在大消息情况下加速了3. plenty倍，从而在LLaMA-2模型上实现了高达1.74倍的TTFT加速和1.34倍的TPOT加速。
英文摘要：
Tensor parallelism (TP) has become a key technique for latency-sensitive LLM inference, but it introduces frequent, tightly synchronized All-Reduce operations that lie directly on the inference critical path, making inter-GPU communication a major performance bottleneck. In-network computing offers a promising approach to mitigating this bottleneck. However, we observe that the existing solution, NVLink SHARP (NVLS), is fundamentally constrained by its GPU-driven execution model that exposes in-network computing as GPU-issued, element-granular memory operations rather than a switch-orchestrated collective. This design forces redundant switch-GPU-switch data movement and cannot perform scale-aware quantized All-Reduce due to its lack of cross-element coordination. To address these limitations, we propose SCIN, the first switch-centric in-network computing architecture for multi-accelerator shared-memory networks. SCIN replaces GPU-driven collective orchestration with an in-switch accelerator (ISA) that directly initiates memory transactions and orchestrates the complete All-Reduce operation. This switch-driven execution eliminates redundant data movement inherent in NVLS, shortens synchronization paths, and fully offloads collective execution from GPUs. Furthermore, centralized ISA control enables cross-element coordination between quantized data and their scale factors. SCIN exploits this capability through a pipelined dequantization-reduction-requantization datapath, enabling scale-aware in-network quantized All-Reduce and nearly halving communication volume with negligible accuracy degradation. We implement SCIN on a multi-FPGA prototype to validate its feasibility. For a simulated 8-GPU system, SCIN accelerates small- and large-message All-Reduce by up to 1.8x and 2.6x over NVLS, respectively, translating into up to 1.12x TPOT and 1.42x TTFT speedups.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
