将大规模电子显微数据流式传输至超级计算设施
Streaming Large-Scale Electron Microscopy Data to a Supercomputing Facility
AI总结：
针对4D相机海量数据存储与处理难题，本文提出一种绕过中间文件存储、将数据从NCEM直接流式传输至NERSC超算节点的方案，实验证明其显著缩短数据周转时间并支持实时决策。
摘要：
数据管理是现代实验工作流程的关键组成部分。随着数据生成速率的不断提高，通过传统的基于文件的方法将数据从采集服务器传输到处理服务器变得越来越不切实际。美国国家电子显微中心（NCEM）的4D相机以480 Gbit/s（87,000帧/秒）的标称速率生成数据，在十五秒内即可产生700 GB的数据集。为应对存储和处理如此大量数据所带来的挑战，我们开发了一种流式工作流程，利用高速网络将4D相机的数据采集（DAQ）系统直接连接到美国国家能源研究科学计算中心（NERSC）的超级计算节点，完全绕过中间文件存储。在这项工作中，我们通过一项长达一小时的实验展示了该流式传输管线在生产环境中的有效性，该实验生成了超过10 TB的原始数据，并产出了适用于高级分析的高质量数据集。此外，我们通过对真实用户所执行实验的历史数据进行事后分析，将该流式工作流程与传统的文件传输工作流程的效能进行了比较。我们的研究结果表明，流式工作流程显著缩短了数据周转时间，实现了实时决策，并通过消除人工用户交互最大限度地减少了人为错误的可能性。
英文摘要：
Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy (NCEM) generates data at a nominal rate of 480 Gbit/s (87,000 frames/s) producing a 700 GB dataset in fifteen seconds. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera's data acquisition (DAQ) system to supercomputing nodes at the National Energy Research Scientific Computing Center (NERSC), bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a post-mortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
