arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1807.01958eess.SPcs.LG
收藏

深度稀疏信号rePresentations($\text{D}\text{S}^2\text{P}$)

Deeply-Sparse Signal rePresentations ($\text{D}\text{S}^2\text{P}$)

Harvard University(哈佛大学)

Demba Ba

摘要:

最近的一项工作表明,具有ReLU非线性的深度神经网络来自级联稀疏编码模型的有限序列,除了级联中的最后一个元素外,其输出是稀疏且不可观测的。也就是说,在级联深处的中间输出是稀疏的,因此这篇手稿的标题。我们在这里表明,使用来自字典学习文献的技术,如果级联稀疏编码模型中的测量矩阵(a)满足RIP并且(b)除了最后一个之外都具有稀疏列,则它们可以以高概率被恢复。为此,我们提出了两种算法:一种是在一个向前的序列中恢复矩阵,另一种是在一个向后的序列中恢复它们。深度学习解决这个问题的方法是训练自动编码器。我们的算法提供了一个健全的替代方案,理论上的保证,以及样本复杂性的上限。该理论表明,前向算法的学习复杂度取决于最深层的隐藏单元的数量和该层的活动神经元的数量(稀疏性)。此外,该理论将连续层中隐藏单元的数量联系起来,从而为设计深度ReLU神经网络提供了实用的处方。因为它对架构的限制较少,所以向后算法需要更多的数据。我们通过模拟演示了深度字典学习算法。最后,我们使用优惠券收集参数来推测样本复杂度的下限,从而深入了解为什么深度网络需要比浅层网络更多的数据来进行训练。

英文摘要:

A recent line of work shows that a deep neural network with ReLU nonlinearities arises from a finite sequence of cascaded sparse coding models, the outputs of which, except for the last element in the cascade, are sparse and unobservable. That is, intermediate outputs deep in the cascade are sparse, hence the title of this manuscript. We show here, using techniques from the dictionary learning literature that, if the measurement matrices in the cascaded sparse coding model (a) satisfy RIP and (b) all have sparse columns except for the last, they can be recovered with high probability. We propose two algorithms for this purpose: one that recovers the matrices in a forward sequence, and another that recovers them in a backward sequence. The method of choice in deep learning to solve this problem is by training an auto-encoder. Our algorithms provide a sound alternative, with theoretical guarantees, as well upper bounds on sample complexity. The theory shows that the learning complexity of the forward algorithm depends on the number of hidden units at the deepest layer and the number of active neurons at that layer (sparsity). In addition, the theory relates the number of hidden units in successive layers, thus giving a practical prescription for designing deep ReLU neural networks. Because it puts fewer restrictions on the architecture, the backward algorithm requires more data. We demonstrate the deep dictionary learning algorithm via simulations. Finally, we use a coupon-collection argument to conjecture a lower bound on sample complexity that gives some insight as to why deep networks require more data to train than shallow ones.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑