数据中心以太网与RDMA：超大规模环境下的问题

面向QP可扩展的超大规模AI训练的通信语义感知RDMA丢包恢复

Communication-Semantic-Aware RDMA Loss Recovery for QP-scalable Hyperscale AI Training

针对RDMA RC模型在超大规模AI训练中因QP数量受限导致的可扩展性差和尾延迟问题，提出CSA-UD机制，解耦数据传输与丢包恢复，动态调整检测间隔，支持多路径传输，显著降低尾延迟并提升可扩展性。

Revisiting Network Support for RDMA

The advent of RoCE (RDMA over Converged Ethernet) has led to a significant increase in the use of RDMA in datacenter networks. To achieve good performance, RoCE requires a lossless network which is in turn achieved by enabling Priority Flow Control (PFC) within the network. However, PFC brings with it a host of problems such as head-of-the-line blocking, congestion spreading, and occasional deadlocks. Rather than seek to fix these issues, we instead ask: is PFC fundamentally required to support RDMA over Ethernet? We show that the need for PFC is an artifact of current RoCE NIC designs rather …

PCM：用于太比特级RDMA网卡的可编程拥塞管理

PCM: Programmable Congestion Management for Terabit RDMA NICs

本文提出PCM编程模型，分离数据中心网卡的可移植拥塞管理与平台集成，实现11种算法并在4种数据路径验证，为Ultra Ethernet可移植拥塞管理接口提供参考。

NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration

We present NetReduce, a novel RDMA-compatible in-network reduction architecture to accelerate distributed DNN training. Compared to existing designs, NetReduce maintains a reliable connection between end-hosts in the Ethernet and does not terminate the connection in the network. The advantage of doing so is that we can fully reuse the designs of congestion control and reliability in RoCE. In the meanwhile, we do not need to implement a high-cost network protocol processing stack in the switch, as IB does. The prototype implemented by using FPGA is an out-of-box solution without modifying commo…

An In-Depth Analysis of the Slingshot Interconnect

The interconnect is one of the most critical components in large scale computing systems, and its impact on the performance of applications is going to increase with the system size. In this paper, we will describe Slingshot, an interconnection network for large scale computing systems. Slingshot is based on high-radix switches, which allow building exascale and hyperscale datacenters networks with at most three switch-to-switch hops. Moreover, Slingshot provides efficient adaptive routing and congestion control algorithms, and highly tunable traffic classes. Slingshot uses an optimized Ethern…

Impact of RoCE Congestion Control Policies on Distributed Training of DNNs

RDMA over Converged Ethernet (RoCE) has gained significant attraction for datacenter networks due to its compatibility with conventional Ethernet-based fabric. However, the RDMA protocol is efficient only on (nearly) lossless networks, emphasizing the vital role of congestion control on RoCE networks. Unfortunately, the native RoCE congestion control scheme, based on Priority Flow Control (PFC), suffers from many drawbacks such as unfairness, head-of-line-blocking, and deadlock. Therefore, in recent years many schemes have been proposed to provide additional congestion control for RoCE network…