arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2102.07725cs.LG
收藏

有噪声存储设备的神经网络压缩

Neural Network Compression for Noisy Storage Devices

Stanford University(斯坦福大学);Meta Reality Labs Research(Meta现实实验室)

Berivan Isik, Kristy Choi, Xin Zheng, Tsachy Weissman, Stefano Ermon, H. -S. Philip Wong, Armin Alaghi

摘要:

神经网络(NN)参数的压缩和有效存储对于在资源受限设备上运行的应用程序至关重要。尽管在NN模型压缩方面取得了重大进展,但在NN参数的实际\textit{physical}存储方面的研究却少得多。传统上,模型压缩和物理存储是解耦的,因为具有纠错码(ECC)的数字存储介质提供鲁棒的无错误存储。然而,这种解耦的方法是低效的,因为它忽略了大多数NN中存在的过参数化,并迫使存储器设备向每个信息位分配相同数量的资源,而不管其重要性如何。在这项工作中,我们调查模拟存储设备作为数字媒体的替代品-一个自然提供了一种方法来增加更多的保护重要位不同于它的对应物,但嘈杂,如果天真地使用,可能会损害存储模型的性能。我们开发了各种强大的编码策略NN的权重存储在模拟设备上,并提出了一种方法来联合优化模型压缩和内存资源分配。然后,我们展示了我们的方法在MNIST,CIFAR-10和ImageNet数据集上训练的模型上对现有压缩技术的有效性。与传统的无错误数字存储相比,我们的方法减少了一个数量级的内存占用,而不会显着损害存储模型的准确性。

英文摘要:

Compression and efficient storage of neural network (NN) parameters is critical for applications that run on resource-constrained devices. Despite the significant progress in NN model compression, there has been considerably less investigation in the actual \textit{physical} storage of NN parameters. Conventionally, model compression and physical storage are decoupled, as digital storage media with error-correcting codes (ECCs) provide robust error-free storage. However, this decoupled approach is inefficient as it ignores the overparameterization present in most NNs and forces the memory device to allocate the same amount of resources to every bit of information regardless of its importance. In this work, we investigate analog memory devices as an alternative to digital media -- one that naturally provides a way to add more protection for significant bits unlike its counterpart, but is noisy and may compromise the stored model's performance if used naively. We develop a variety of robust coding strategies for NN weight storage on analog devices, and propose an approach to jointly optimize model compression and memory resource allocation. We then demonstrate the efficacy of our approach on models trained on MNIST, CIFAR-10 and ImageNet datasets for existing compression techniques. Compared to conventional error-free digital storage, our method reduces the memory footprint by up to one order of magnitude, without significantly compromising the stored model's accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑