LegoNet:通过块权重聚类减少内存占用
LegoNet: Memory Footprint Reduction Through Block Weight Clustering
AI总结:
LegoNet通过块权重聚类技术,在不改变架构和不重新训练的情况下,实现高达128倍的内存压缩,同时保持接近97%的精度。
摘要:
随着神经网络应用对精度和性能需求的增加,其规模和内存占用也在增长。在嵌入式设备中,由于缓存和RAM有限,这种增长阻碍了它们利用最新神经网络架构的能力。在本工作中,我们提出了LegoNet,一种压缩技术,该技术无论层类型如何,都构建整个模型的所有权重块,并对这些诱导块进行聚类。使用块而不是单个值来聚类权重,我们能够仅用32个4x4块压缩训练了Cifar-10和ImageNet的ResNet-50,将内存占用压缩了超过64倍,而无需删除任何权重或改变架构,并且没有损失精度,也不需要重新训练或任何数据,并展示了如何找到16个4x4块的排列,以获得128倍的压缩比,同时仅损失不到3%的精度。这全部是在不需要(重新)训练或微调的情况下完成的。
英文摘要:
As the need for neural network-based applications to become more accurate and powerful grows, so too does their size and memory footprint. With embedded devices, whose cache and RAM are limited, this growth hinders their ability to leverage state-of-the-art neural network architectures. In this work, we propose \textbf{LegoNet}, a compression technique that \textbf{constructs blocks of weights of the entire model regardless of layer type} and clusters these induced blocks. Using blocks instead of individual values to cluster the weights, we were able to compress ResNet-50 trained for Cifar-10 and ImageNet with only 32 4x4 blocks, compressing the memory footprint by over a factor of \textbf{64x without having to remove any weights} or changing the architecture and \textbf{no loss to accuracy}, nor retraining or any data, and show how to find an arrangement of 16 4x4 blocks that gives a compression ratio of \textbf{128x with less than 3\% accuracy loss}. This was all achieved with \textbf{no need for (re)training or fine-tuning}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。