使深度神经网络紧凑的分支
The Ramifications of Making Deep Neural Networks Compact
摘要：
深度神经网络（DNN）研究的最新趋势是使网络更加紧凑。设计紧凑型DNN背后的动机是提高能量效率，因为凭借具有较低的存储器占用，紧凑型DNN具有较低数量的片外访问，这提高了能量效率。然而，我们表明，使DNN紧凑有间接和微妙的影响，这是不好理解的。减少DNN中的参数数量会增加激活次数，这反过来又会增加内存占用。我们在Tesla P100 GPU上评估了几个最近提出的紧凑型DNN，并显示它们的“激活参数比”范围在1.4到32.8之间。此外，“内存占用与模型大小的比率”范围在15到443之间。这表明，较高的激活次数会导致较大的内存占用，从而增加片内/片外数据移动。此外，这些参数减少技术降低了增加片上/片外内存带宽要求的运算强度。由于这些因素，紧凑型DNN的能量效率可能会显著降低，这与设计紧凑型DNN的原始动机相反。
英文摘要：
The recent trend in deep neural networks (DNNs) research is to make the networks more compact. The motivation behind designing compact DNNs is to improve energy efficiency since by virtue of having lower memory footprint, compact DNNs have lower number of off-chip accesses which improves energy efficiency. However, we show that making DNNs compact has indirect and subtle implications which are not well-understood. Reducing the number of parameters in DNNs increases the number of activations which, in turn, increases the memory footprint. We evaluate several recently-proposed compact DNNs on Tesla P100 GPU and show that their "activations to parameters ratio" ranges between 1.4 to 32.8. Further, the "memory-footprint to model size ratio" ranges between 15 to 443. This shows that a higher number of activations causes large memory footprint which increases on-chip/off-chip data movements. Furthermore, these parameter-reducing techniques reduce the arithmetic intensity which increases on-chip/off-chip memory bandwidth requirement. Due to these factors, the energy efficiency of compact DNNs may be significantly reduced which is against the original motivation for designing compact DNNs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
