RED-ONION：高速磁盘到磁盘传输系统的性能评估
Performance Evaluation of RED-ONION: A High-Speed Disk-to-Disk Transfer System
AI总结：
针对实验数据传输瓶颈，提出RED-ONION高速磁盘到磁盘传输系统，通过多组件协同优化，在跨太平洋路径实现单1TB文件90Gbps传输，可实现仪器与计算中心的同址化服务。
AI中文摘要：
现代实验仪器产生数据的速度远超通用文件传输接口的移动速度，因此将数据传输至计算基础设施已成为研究过程中的瓶颈。此外，在许多大学和研究机构中，生成研究数据的仪器与分析数据的高性能计算系统在地理和组织上相互分离，因为两者都需要各自的专业知识和安装环境，将两者无缝连接是数据驱动科学面临的紧迫挑战。本文提出RED-ONION，一种高速磁盘到磁盘传输系统，可将校园及其他地区的研究设施连接至计算中心。该系统结合了数据传输节点、专用高带宽网络、全闪存并行文件系统，以及可并行化网络传输和存储访问的多线程传输软件。其设计目标是实现整个路径（从发送方存储读取到接收方存储写入）以及单个节点对之间单个文件的线速，而非仅在多个文件或节点上的聚合线速。我们描述了该系统在传输软件、操作系统和存储方面所需的端到端优化。我们评估了部署在亚特兰大与东京之间100 Gbps跨太平洋路径上的原型，该路径往返时间为150 ms，在该路径上单个1 TB文件的传输速度达到90 Gbps，约95秒即可传输1 TB数据。因此，移动此规模的数据集成为常规操作，计算中心为仪器提供服务时，就如同两者位于同一地点。
英文摘要：
Modern experimental instruments produce data faster than general-purpose file transfer interfaces can move it, so delivery to the computing infrastructure has become a bottleneck in the research process. At many universities and research institutes, moreover, the instruments that generate research data and the high-performance computing systems that analyze it are separated both geographically and organizationally, because each demands its own expertise and installation environment. Connecting the two seamlessly is a pressing challenge for data-driven science. This article presents RED-ONION, a high-speed disk-to-disk transfer system that connects research facilities, on campus and beyond, to a computing center. The system combines data transfer nodes, a dedicated high-bandwidth network, an all-flash parallel file system, and multi-threaded transfer software that parallelizes network transmission and storage access. The design targets the wire rate both along the entire path, from the read on the sender storage to the write on the receiver storage, and for a single file between one pair of nodes rather than only in aggregate over many files or nodes. We describe the end-to-end optimizations across the transfer software, the operating system, and the storage that this requires. We evaluate a prototype deployed over a 100 Gbps transpacific path between Atlanta and Tokyo with a 150 ms round-trip time, on which a single 1 TB file transfer reached 90 Gbps, delivering a terabyte in approximately 95 s. Moving a dataset of this size therefore becomes a routine step, and the computing center serves an instrument as if the two were co-located.
