使用加速关联（令牌）数组分析10 Petabit/s网络数据
Analyzing 10 Petabit/s Network Data with Accelerated Associative (Token) Arrays
浏览论文内容
中文总结 AI 辅助
针对大规模网络隐私分析需求，利用GPU加速的D4M关联数组库，在多种硬件上实现水平线性扩展，达到可分析10 Petabit/s网络的持续处理速率。
中文摘要 AI 辅助
随着网络不断扩展并成为现代社会日益关键的基础设施，以最高隐私标准分析这些网络对于确保其正常运行至关重要。根据待分析的网络层级别，源和目的地可以是物理、逻辑或人/代理端点的任意组合，这要求能够处理多样化的数据。对这些分析而言，数学工具至关重要，它们能够简洁地表达复杂的数学算法，同时实现可扩展的垂直（在计算节点内）、水平（跨计算节点）和时间（跨不同代硬件）性能。关联（令牌）数组数学及相应库是满足这些要求的一种方法。使用GPU加速这些库能够分析最大的网络。MIT/IEEE/Amazon匿名网络感知图挑战为展示加速关联数组在这类问题中的适用性提供了平台。D4M关联库已用多种语言实现。本研究在广泛的CPU和GPU硬件上，对匿名网络感知挑战的原型Matlab D4M GPU加速实现进行了基准测试。在CPU核心内和跨CPU核心、CPU节点和GPU节点均展示了可扩展性能。跨多个节点的水平扩展是线性的。同时在数百个GPU节点上运行，实现了持续处理速率，足以潜在地分析10 Petabit/s的网络。
英文摘要
As networks expand and become an ever more critical infrastructure to modern society the need to analyze these networks with the highest regard for privacy is essential to ensure their proper function. Depending on the level of the network layer to be analyzed, sources and destinations can be any combination of physical, logical, or persona/agentic endpoints, which requires the ability to handle diverse data. Invaluable to these analyses are mathematical tools that enable sophisticated mathematical algorithms to be expressed succinctly while achieving scalable vertical (within a compute node), horizontal (across compute nodes), and temporal (over different generations of hardware) performance. Associative (token) array mathematics and corresponding libraries is one approach that can meet these requirements. Accelerating these libraries with GPUs enables the analysis of the largest networks. The MIT/IEEE/Amazon Anonymized Network Sensing Graph Challenge provides a venue for highlighting the applicability of accelerated associative arrays for these types of problems. The D4M associative library has been implemented in a number of languages. This work benchmarks a prototype Matlab D4M GPU accelerated implementation of the Anonymized Network Sensing challenge across a wide range of CPU and GPU hardware. Scalable performance is demonstrated within and across CPU cores, CPU nodes, and GPU nodes. Horizontal scaling across multiple nodes was linear. Running on hundreds of GPU nodes simultaneously achieved a sustained processing rate sufficient to potentially analyze a 10 Petabit/s network.
发表机构
- MIT（麻省理工学院）
机构由 AI 辅助整理，请以论文原文为准。
