arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1707.09320cs.OHastro-ph.IMcs.CE
收藏

Z-checker:科学数据有损压缩评估框架

Z-checker: A Framework for Assessing Lossy Compression of Scientific Data

Dingwen Tao, Sheng Di, Hanqi Guo, Zizhong Chen, Franck Cappello

摘要:

由于当今的科学模拟和实验产生了大量的数据,因此在压缩期间允许用户控制准确性损失的有损数据压缩器是显著减小数据大小的相关解决方案。然而,有损压缩器的开发人员和用户缺少一个工具来探索科学数据集的特征,并以系统和可靠的方式了解压缩后的数据更改。为了解决这个问题,我们设计并实现了一个通用框架,称为Z-checker。一方面,Z-checker结合了一组数据分析组件,用于数据压缩。另一方面,Z-checker是一个开源社区工具,用户和开发人员可以根据他们的额外分析需求贡献和添加新的分析组件。在本文中,我们提出了现有的有损压缩机的调查。然后,我们描述了Z-检查器的设计框架,在其中,我们集成了以前的工作中提出的评估指标以及其他分析工具。具体而言,对于有损压缩器开发人员,Z检查器可用于表征任何数据集的关键属性,以改进压缩策略。对于有损压缩用户,Z-checker可以检测压缩质量,提供各种全局失真分析,将原始数据与解压缩数据进行比较,并对压缩误差进行统计分析。Z-checker可以以粗粒度或细粒度执行分析,这样用户和开发人员就可以为数据集的不同部分选择最适合的自适应压缩器。Z-checker具有一个可视化界面,除了时间序列等数据集的一些基本视图之外,还显示所有分析结果。据我们所知,Z-checker是第一个旨在全面评估科学数据集有损压缩的工具。

英文摘要:

Because of vast volume of data being produced by today's scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific datasets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this paper, we present a survey of existing lossy compressors. Then we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties of any dataset to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality, provide various global distortion analysis comparing the original data with the decompressed data and statistical analysis of the compression error. Z-checker can perform the analysis with either coarse granularity or fine granularity, such that the users and developers can select the best-fit, adaptive compressors for different parts of the dataset. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the datasets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑