二次深度网络的通用逼近
Universal Approximation with Quadratic Deep Networks
摘要:
最近,深度学习在许多重要的应用中取得了巨大的成功。在我们以前的研究中,我们提出了二次/二阶神经元和深度二次神经网络。在二次神经元中,传统神经元中的数据向量和相应权重的内积被二次函数取代。由此产生的二次神经元比传统神经元具有更强的表达能力。然而,二次神经元如何提高深度二次网络的表达能力迄今为止还没有被研究,优选地与传统神经网络的表达能力相关。关于这一点,我们在本文中提出了四个基本问题:(1)对于一个隐藏层的网络结构,是否有任何功能,一个二次网络可以更有效地近似比传统的网络?(2)对于同样的多层网络结构,是否存在可以用二次网络表示而不能用同样结构的传统神经元表示的函数?(3)一个二次网络是否给了一个新的洞察力,以普遍的近似?(4)为了近似具有相同误差界的同类函数,量化的二次网络是否能够比量化的传统网络享有更少的权重?我们的主要贡献是四个相互关联的定理,揭示了这四个问题,并展示了二次网络的优点,表达效率,独特的能力,紧凑的架构和计算能力。
英文摘要:
Recently, deep learning has achieved huge successes in many important applications. In our previous studies, we proposed quadratic/second-order neurons and deep quadratic neural networks. In a quadratic neuron, the inner product of a vector of data and the corresponding weights in a conventional neuron is replaced with a quadratic function. The resultant quadratic neuron enjoys an enhanced expressive capability over the conventional neuron. However, how quadratic neurons improve the expressing capability of a deep quadratic network has not been studied up to now, preferably in relation to that of a conventional neural network. Regarding this, we ask four basic questions in this paper: (1) for the one-hidden-layer network structure, is there any function that a quadratic network can approximate much more efficiently than a conventional network? (2) for the same multi-layer network structure, is there any function that can be expressed by a quadratic network but cannot be expressed with conventional neurons in the same structure? (3) Does a quadratic network give a new insight into universal approximation? (4) To approximate the same class of functions with the same error bound, is a quantized quadratic network able to enjoy a lower number of weights than a quantized conventional network? Our main contributions are the four interconnected theorems shedding light upon these four questions and demonstrating the merits of a quadratic network in terms of expressive efficiency, unique capability, compact architecture and computational capacity respectively.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。