Deep neural networks (DNNs) have been widely deployed across diverse domains such as computer vision and natural language processing. However, the impressive accomplishments of DNNs have been realized alongside extensive computational demands, thereby impeding their applicability on resource-constrained devices. To address this challenge, many researchers have been focusing on basic neuron structures, the fundamental building blocks of neural networks, to alleviate the computational and storage cost. In this work, an efficient quadratic neuron architecture distinguished by its enhanced utilization of second-order computational information is introduced. By virtue of their better expressivity, DNNs employing the proposed quadratic neurons can attain similar accuracy with fewer neurons and computational cost. Experimental results have demonstrated that the proposed quadratic neuron structure exhibits superior computational and storage efficiency across various tasks when compared with both linear and non-linear neurons in prior work.
Reconstructing Spiking Neural Networks Using a Single Neuron with Autapses
Brain-Apparatus Communication Institute, University of Electronic Science and Technology of China(脑器通信研究院,电子科技大学);School of Artificial Intelligence, Chongqing University of Education(人工智能学院,重庆教育学院)
One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks
Chinese University of Hong Kong(香港中文大学);Harbin Institute of Technology(哈尔滨工业大学);University of Florida(佛罗里达大学);Beijing Institute for General Artificial Intelligence(北京通用人工智能研究院)
受生物神经系统中神经元多样性的启发,大量的研究提出设计新型的人工神经元,并将神经元多样性引入人工神经网络。最近提出的二次神经元,它取代了传统的神经元的内积运算与二次之一,取得了巨大的成功,在许多重要的任务。尽管二次神经元的结果很有希望,但仍然有一个未解决的问题:\textit{Is the superior performance of quadratic networks simply due to the increased parameters or due to the intrinsic expressive capability?}在没有澄清这个问题的情况下,二次网络的性能总是可疑的。此外,解决这个问题可以归结为寻找二次网络的杀手级应用。本文通过理论和实证研究,证明了二次网络具有参数有效性,从而证实了二次网络的优越性能是由于其内在的表达能力。这种内在的表达能力来自于二次神经元可以很容易地表示非线性相互作用,而传统的神经元很难。理论上,我们推导了二次网络在实空间和流形上的逼近效率。此外,从Barron空间的角度来看,我们证明了存在一个功能空间,其功能可以近似的二次网络的无量纲误差,但传统的网络的近似误差是依赖于尺寸。从经验上讲,对合成数据、经典基准测试和现实应用的实验结果表明,二次模型广泛享有参数效率,效率的提高取决于任务。
Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detection
Harbin Institute of Technology(哈尔滨工业大学);The Hong Kong Polytechnic University(香港理工大学);Weill Cornell Medicine(威尔康奈尔医学院);Cornell University(康奈尔大学);The Chinese University of Hong Kong(香港中文大学)
Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent
University of Bonn(波恩大学);University of Tübingen(图宾根大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Discretized Quadratic Integrate-and-Fire Neuron Model for Deep Spiking Neural Networks
STAM Center Arizona State University(STAM中心亚利桑那州立大学);Center for Advanced Studies and Systems of Recife(里贝拉先进研究与系统中心);Faculty of Technical Sciences University of Novi Sad(诺维萨德大学技术科学学院)
Neurons Merging Layer: Towards Progressive Redundancy Reduction for Deep Supervised Hashing
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Horizon Robotics(地平线机器人);Center for Excellence in Brain Science and Intelligence Technology, CAS(中国科学院脑科学与智能技术卓越创新中心)
Activation function dependence of the storage capacity of treelike neural networks
Harvard University(哈佛大学);John A. Paulson School of Engineering and Applied Sciences, Harvard University(哈佛大学约翰·A·保尔森工程与应用科学学院);Center for Brain Science, Harvard University(哈佛大学脑科学中心)
Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions
ETH Zurich(苏黎世联邦理工学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);University of Münster(明斯特大学)
On associative neural networks for sparse patterns with huge capacities
Fachbereich Mathematik und Informatik, University of M\"unster, Einsteinstra e 62, 48149 M\"unster, Germany;Univ Brest, CNRS UMR 6205, Laboratoire de Math\'ematiques de Bretagne Atlantique, 6, avenue Victor Le Gorgeu CS 93837 F-29238 BREST Cedex 3 France