arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.13429cs.DBcs.LGcs.OS
收藏

关于预训练机器学习模型存储可压缩性你一直想知道却不敢问的一切

Everything You Always Wanted to Know About Storage Compressibility of Pre-Trained ML Models but Were Afraid to Ask

University of Virginia(弗吉尼亚大学);University of Minnesota(明尼苏达大学)

Zhaoyuan Su, Ammar Ahmed, Zirui Wang, Ali Anwar, Yue Cheng

AI总结:

针对预训练模型存储压缩需求,本文首次系统分析其存储可压缩性,提出ELF有损浮点压缩方法与Elves框架,实现1.52×压缩比且精度损失可忽略。

摘要:

随着预训练机器学习(ML)模型的数量呈指数级增长,数据缩减工具的发展未能跟上步伐。现有的数据缩减技术并非专门针对预训练模型(PTM)数据集文件设计,这在很大程度上是因为人们对这类数据集的模式与特征缺乏了解,尤其是与数据缩减和可压缩性相关的特征。\n 本文首次对PTM数据集的存储可压缩性开展了迄今为止最详尽的分析。我们的分析覆盖了不同类型的数据缩减与压缩技术,从基于哈希的数据去重、数据相似性检测到字典编码压缩。我们在三个数据粒度层面探究了这些技术,分别为模型层、模型块与模型参数。我们得出的新结论表明,现有数据缩减工具在处理PTM数据集时效果不佳,迫切需要结合PTM数据特征的新型压缩方法,以实现高效的存储缩减。\n 基于上述发现,我们设计了ELF——一种简单却有效的带误差界的有损浮点压缩方法。ELF对浮点参数进行变换,使得变换后参数的公共指数域可被完全消除,从而节省存储空间。我们开发了Elves压缩框架,该框架整合了ELF与其他多种数据缩减方法,能够针对呈现不同模式的PTM选用最高效的压缩方法。评估结果显示,Elves的整体压缩比达到1.52×,分别比通用压缩器(zstd)、带误差界的有损压缩器(SZ3)和均匀模型量化高1.31×、1.32×与1.29×,且模型精度损失可忽略不计。

英文摘要:

As the number of pre-trained machine learning (ML) models is growing exponentially, data reduction tools are not catching up. Existing data reduction techniques are not specifically designed for pre-trained model (PTM) dataset files. This is largely due to a lack of understanding of the patterns and characteristics of these datasets, especially those relevant to data reduction and compressibility. This paper presents the first, exhaustive analysis to date of PTM datasets on storage compressibility. Our analysis spans different types of data reduction and compression techniques, from hash-based data deduplication, data similarity detection, to dictionary-coding compression. Our analysis explores these techniques at three data granularity levels, from model layers, model chunks, to model parameters. We draw new observations that indicate that modern data reduction tools are not effective when handling PTM datasets. There is a pressing need for new compression methods that take into account PTMs' data characteristics for effective storage reduction. Motivated by our findings, we design ELF, a simple yet effective, error-bounded, lossy floating-point compression method. ELF transforms floating-point parameters in such a way that the common exponent field of the transformed parameters can be completely eliminated to save storage space. We develop Elves, a compression framework that integrates ELF along with several other data reduction methods. Elves uses the most effective method to compress PTMs that exhibit different patterns. Evaluation shows that Elves achieves an overall compression ratio of $1.52\times$, which is $1.31\times$, $1.32\times$ and $1.29\times$ higher than a general-purpose compressor (zstd), an error-bounded lossy compressor (SZ3), and the uniform model quantization, respectively, with negligible model accuracy loss.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑