深度学习中条件计算的计算能力比呈指数增长
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
摘要：
深度网络获得的许多最先进的结果都是通过可以训练的最大模型实现的，如果有更多的计算能力，我们可能能够利用更大的数据集来提高泛化能力。而在诸如决策树之类的学习算法中，参数的数量）与计算的比率是非常有利的（高达指数地多于计算的参数），对于深度神经网络，该比率基本上为1。条件计算已经被提出作为一种在不增加所需计算量的情况下增加深度神经网络的容量的方式，其通过在每个示例的基础上“按需”激活一些参数和计算。本文提出了一种新的神经网络权值矩阵参数化方法，它有可能使计算参数数的比率以指数方式增加。所提出的方法是基于打开一些参数（权重矩阵）时，获得特定的位模式的隐藏单元激活。为了更好地控制可能导致的过拟合，我们提出了一个参数化，是树结构的，其中树的每个节点对应于一个前缀的一个序列的符号位，或门控单元，与隐藏单元。
英文摘要：
Many state-of-the-art results obtained with deep networks are achieved with the largest models that could be trained, and if more computation power was available, we might be able to exploit much larger datasets in order to improve generalization ability. Whereas in learning algorithms such as decision trees the ratio of capacity (e.g., the number of parameters) to computation is very favorable (up to exponentially more parameters than computation), the ratio is essentially 1 for deep neural networks. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation required, by activating some parameters and computation "on-demand", on a per-example basis. In this note, we propose a novel parametrization of weight matrices in neural networks which has the potential to increase up to exponentially the ratio of the number of parameters to computation. The proposed approach is based on turning on some parameters (weight matrices) when specific bit patterns of hidden unit activations are obtained. In order to better control for the overfitting that might result, we propose a parametrization that is tree-structured, where each node of the tree corresponds to a prefix of a sequence of sign bits, or gating units, associated with hidden units.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
