AI 中文总结
该研究针对PETRA III数据集开展综合无损压缩性能研究，评估九种压缩工具，发现异构压缩策略更优，为大型科研设施存储决策提供定量依据。
AI 中文摘要
大型科研设施日益面临着在维持可持续归档基础设施的同时管理快速增长的数据量的挑战。我们针对PETRA III同步辐射源的代表性数据集，开展了首个关于数据异质性和通用无损压缩性能的综合研究。我们的语料库包含来自十个实验的超过212 TiB的原始及处理后数据，这些实验覆盖多个光束线、探测器系统和科学工作流程。我们观察到实验之间及实验内部存在显著异质性，导致不同数据集的压缩比相差两个数量级以上。通过评估九种广泛使用的无损压缩工具，我们发现Zstandard和LZ4始终处于帕累托前沿的高吞吐量区域，而ZPAQ实现了最高的压缩比。此外，根据底层文件类别调整压缩器选择的异构压缩策略，优于统一的压缩策略。从基准测试数据集外推至完整的PETRA III非磁带存储系统，我们估计可实现的压缩比范围为：在约900 MiB/s吞吐量下约为1.6，在约2 MiB/s吞吐量下约为2.1。这些结果为PETRA III及其未来继承者PETRA IV和其他大型科研设施的未来归档和存储决策提供了定量依据。
英文摘要
Large-scale research facilities increasingly face the challenge of managing rapidly growing data volumes while maintaining sustainable archival infrastructures. We present the first comprehensive study of data heterogeneity and lossless general-purpose compression performance for representative datasets from the PETRA III synchrotron radiation source. Our corpus comprises more than 212 TiB of raw and processed data from ten experiments spanning multiple beamlines, detector systems, and scientific workflows. We observe substantial heterogeneity both between and within experiments, resulting in compression ratios that vary by more than two orders of magnitude across datasets. Evaluating nine widely used lossless compression tools, we find that Zstandard and LZ4 consistently occupy the high-throughput region of the Pareto front, whereas ZPAQ achieves the highest compression ratios. Furthermore, heterogeneous compression strategies that adapt compressor choice to the underlying file category outperform uniform compression policies. Extrapolating from the benchmarked datasets to the full PETRA III non-tape storage system, we estimate achievable compression ratios ranging from approximately 1.6 at $\sim$900 MiB/s throughput to 2.1 at $\sim$2 MiB/s. These results provide a quantitative basis for future archival and storage decisions at PETRA III, its future successor, PETRA IV, and other large-scale scientific facilities.
Comments18+7 pages, 9+0 figures, 5+9 tables
