arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.03201cs.LGstat.ML
收藏

深度神经网络的可微稀疏化

Differentiable Sparsification for Deep Neural Networks

ETRI(韩国电子通信研究院)

Yognjin Lee

AI总结:

针对深度神经网络架构设计繁琐及模型规模过大的问题,提出一种完全可微的稀疏化方法,通过随机梯度下降直接优化正则化目标函数将不重要参数置零,实现端到端学习网络稀疏结构与权重,且仅需极少修改即可应用于现代网络。

摘要:

深度神经网络显著减轻了特征工程的负担,但现在需要付出相当的努力来确定这些网络的有效架构。此外,随着网络规模变得过大,大量资源被投入到缩减其规模上。这些挑战可以通过对过完备模型进行稀疏化来有效解决。在本研究中,我们提出了一种用于深度神经网络的完全可微的稀疏化方法,该方法可以通过使用随机梯度下降直接优化正则化目标函数来将不重要的参数置零。因此,所提出的方法能够以端到端的方式学习网络的稀疏化结构和权重。它可以直接应用于各种现代深度神经网络,并且只需对训练过程进行极少的修改。据我们所知,这是首个完全可微的稀疏化方法。

英文摘要:

Deep neural networks have significantly alleviated the burden of feature engineering, but comparable efforts are now required to determine effective architectures for these networks. Furthermore, as network sizes have become excessively large, a substantial amount of resources is invested in reducing their sizes. These challenges can be effectively addressed through the sparsification of over-complete models. In this study, we propose a fully differentiable sparsification method for deep neural networks, which can zero out unimportant parameters by directly optimizing a regularized objective function with stochastic gradient descent. Consequently, the proposed method can learn both the sparsified structure and weights of a network in an end-to-end manner. It can be directly applied to various modern deep neural networks and requires minimal modification to the training process. To the best of our knowledge, this is the first fully differentiable sparsification method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑