TuckerMPI：基于Tucker张量分解的大规模数据压缩并行C++/ MPI软件包
TuckerMPI: A Parallel C++/MPI Software Package for Large-scale Data Compression via the Tucker Tensor Decomposition
摘要：
我们的目标是压缩大规模的网格结构数据，例如高保真计算模拟的多TB输出。对于这样的数据集，我们已经开发了一个新的软件包TuckerMPI，一个并行的C++/MPI软件包压缩分布式数据。该方法基于将数据视为张量，即，一个多维数组，并计算其截断塔克分解，高阶类似于矩阵的截断奇异值分解。结果是原始张量结构数据的低秩近似。压缩效率是通过检测数据中潜在的全局结构来实现的，这与大多数集中在局部结构上的压缩方法不同。在这项工作中，我们描述了TuckerMPI，我们的截断塔克分解的实现，包括数据分布和内存布局的细节，并行和串行实现的关键内核，以及分析的存储，通信和计算成本。我们在分布在100个节点（1000个MPI进程）上的4.5 TB和6.7 TB数据集上测试了该软件，实现了100- 200，000 $\times$之间的压缩率，相当于99-99.999%的压缩率（取决于所需的准确度），所用的时间比从并行文件系统读取相同数据集的时间要少得多。此外，我们表明，我们的方法还允许在单个节点上重建部分或下采样数据，而无需并行计算机，只要重建的部分足够小以适合单个机器，例如，在重建/可视化单个下采样时间步或计算汇总统计量的情况下。
英文摘要：
Our goal is compression of massive-scale grid-structured data, such as the multi-terabyte output of a high-fidelity computational simulation. For such data sets, we have developed a new software package called TuckerMPI, a parallel C++/MPI software package for compressing distributed data. The approach is based on treating the data as a tensor, i.e., a multidimensional array, and computing its truncated Tucker decomposition, a higher-order analogue to the truncated singular value decomposition of a matrix. The result is a low-rank approximation of the original tensor-structured data. Compression efficiency is achieved by detecting latent global structure within the data, which we contrast to most compression methods that are focused on local structure. In this work, we describe TuckerMPI, our implementation of the truncated Tucker decomposition, including details of the data distribution and in-memory layouts, the parallel and serial implementations of the key kernels, and analysis of the storage, communication, and computational costs. We test the software on 4.5 terabyte and 6.7 terabyte data sets distributed across 100s of nodes (1000s of MPI processes), achieving compression rates between 100-200,000$\times$ which equates to 99-99.999% compression (depending on the desired accuracy) in substantially less time than it would take to even read the same dataset from a parallel filesystem. Moreover, we show that our method also allows for reconstruction of partial or down-sampled data on a single node, without a parallel computer so long as the reconstructed portion is small enough to fit on a single machine, e.g., in the instance of reconstructing/visualizing a single down-sampled time step or computing summary statistics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
