Pcodec:数值序列的更好压缩
Pcodec: Better Compression for Numerical Sequences
AI总结:
Pcodec通过分箱算法和两种预处理步骤,在六个真实列式数据集上实现比现有数值编解码器高29-94%的压缩比,且压缩时间更少。
摘要:
我们提出了Pcodec(Pco),一种用于无损压缩数值(浮点数或整数)序列的格式和算法。Pco的核心且最具创新性的组件是一种分箱算法,该算法能快速收敛到平滑、独立且同分布(SIID)整数的真实熵。我们从数学上证明了这种收敛性,并给出了一个实用的界。为了适应非SIID的数据,Pco有两个针对性的预处理步骤。第一步,Pco的模式,将数字分解为分布更平滑的整数潜变量。第二步,增量编码,使潜变量更加独立且同分布。我们证明,在六个真实世界的列式数据集上,Pco实现了比其他数值编解码器高29-94%的压缩比,同时使用的压缩时间更少。
英文摘要:
We present Pcodec (Pco), a format and algorithm for losslessly compressing numerical (float or integer) sequences. Pco's core and most novel component is a binning algorithm that quickly converges to the true entropy of smoothly, independently, and identically distributed (SIID) integers. We mathematically prove this convergence with a practical bound. To accommodate data this is not SIID, Pco has two opinionated preprocessing steps. The first step, Pco's mode, decomposes the numbers into more smoothly distributed integer latent variables. The second step, delta encoding, makes the latents more independently and identically distributed. We demonstrate that Pco achieves 29-94% higher compression ratio than other numerical codecs on six real-world columnar datasets while using less compression time.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。