首页/论文/2007.11136

arXiv 2007.111362020-07cond-mat.dis-nncs.LGstat.ML

Activation function dependence of the storage capacity of treelike neural networks

Harvard University（哈佛大学）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Center for Brain Science, Harvard University（哈佛大学脑科学中心）

Jacob A. Zavatone-Veth, Cengiz Pehlevan

备注：: 5+23 pages, 2+4 figures. v3: accepted for publication as a Letter in Physical Review E
发表信息：: Phys. Rev. E 103, 020301 (2021)
DOI：: 10.1103/PhysRevE.103.L020301

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

The expressive power of artificial neural networks crucially depends on the nonlinearity of their activation functions. Though a wide variety of nonlinear activation functions have been proposed for use in artificial neural networks, a detailed understanding of their role in determining the expressive power of a network has not emerged. Here, we study how activation functions affect the storage capacity of treelike two-layer networks. We relate the boundedness or divergence of the capacity in the infinite-width limit to the smoothness of the activation function, elucidating the relationship between previously studied special cases. Our results show that nonlinearity can both increase capacity and decrease the robustness of classification, and provide simple estimates for the capacity of networks with several commonly used activation functions. Furthermore, they generate a hypothesis for the functional benefit of dendritic spikes in branched neurons.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

具有通用激活函数的全连接两层神经网络的解空间与存储容量

Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions

School of Engineering, Tohoku University（东北大学工学院）；Graduate School of Information Sciences, Tohoku University（东北大学信息科学研究科）；Department of Physics, Tokyo Institute of Technology（东京工业大学物理系）；Sigma-i Co., Ltd.（Sigma-i有限公司）

本研究用统计物理复本方法分析通用激活函数全连接两层神经网络的解空间与存储容量，发现无限宽度下每参数存储容量仍有限、权重负相关分工、数据集增大触发相变致解空间分裂，揭示激活函数对转变点和存储容量的影响。

2024-11-29cond-mat.dis-nncs.LGstat.ML

固定宽度树状神经网络容量分析——通用激活函数

Fixed width treelike neural networks capacity analysis -- generic activations

本文利用随机对偶理论及其部分提升版本，分析树状委员会机器在通用激活函数下的容量，发现容量随隐藏层宽度增大而收敛至常数，且两个隐藏层神经元时容量最大。

2024-02-08cond-mat.dis-nncs.ITcs.LG

宽隐层树状神经网络在通用激活函数下的精确容量

Exact capacity of the \emph{wide} hidden layer treelike neural networks with generic activations

本文利用完全提升随机对偶理论，刻画了宽隐层树状委员会机器网络在通用激活函数下的闭式容量，并验证了其与复制理论结果的一致性。

2024-02-08cond-mat.dis-nncs.ITcs.LG

ADaPTION: Toolbox and Benchmark for Training Convolutional Neural Networks with Reduced Numerical Precision Weights and Activation

Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) are useful for many practical tasks in machine learning. Synaptic weights, as well as neuron activation functions within the deep network are typically stored with high-precision formats, e.g. 32 bit floating point. However, since storage capacity is limited and each memory access consumes power, both storage capacity and memory access are two crucial factors in these networks. Here we present a method and present the ADaPTION toolbox to extend the popular deep learning library Caffe to support training of deep CNNs with redu…

2017-11-13cs.LG

On neural network kernels and the storage capacity problem

Harvard University（哈佛大学）

In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.

2022-01-12cond-mat.dis-nncs.LG

Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning

New York University（纽约大学）

Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on encrypted inputs. PI protects clients' personal data and the server's intellectual property. A common practice in PI is to use garbled circuits to compute nonlinear functions privately, namely ReLUs. However, garbled circuits suffer from high storage, bandwidth, and latency costs. To mitigate these issues, PI-friendly polynomial activation functions have been employed to replace ReLU. In this work, we ask: Is it feasible to substitute all ReLUs with low-degree…

2021-11-02cs.CRcs.LG

更多相关论文收起更多论文

GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm

Zhejiang University（浙江大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. This is due to its significantly less computation and storage demand, but at the cost of degraded performance. To close the accuracy gap, in this paper we propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs. These AFs can help extract extra information from the input data in the forward pass, while allowing improved gradient approximation i…

2022-06-05cs.LGcs.NE

AWEQ：面向大语言模型的激活-权重均衡后训练量化

AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models

School of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）

AWEQ是一种无需额外训练的后训练量化方法，通过通道均衡将激活量化难度转移至权重，在超低位和W8A8量化中实现精度与硬件效率的平衡，并在LLaMA和OPT上优于现有方法。

2023-11-12cs.AIcs.CLcs.LG

Lossy compression of multidimensional medical images using sinusoidal activation networks: an evaluation study

Cardiff University（卡迪夫大学）

In this work, we evaluate how neural networks with periodic activation functions can be leveraged to reliably compress large multidimensional medical image datasets, with proof-of-concept application to 4D diffusion-weighted MRI (dMRI). In the medical imaging landscape, multidimensional MRI is a key area of research for developing biomarkers that are both sensitive and specific to the underlying tissue microstructure. However, the high-dimensional nature of these data poses a challenge in terms of both storage and sharing capabilities and associated costs, requiring appropriate algorithms able…

2022-08-03cs.CVcs.LGeess.IV

DECO：稀疏专家混合模型在端侧设备上实现密集级性能

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices

Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China（计算机科学与技术系，人工智能研究院，清华大学，北京，中国）

本文提出DECO，一种稀疏的专家混合模型，能够在相同的总参数预算和训练token数量下，实现与密集Transformer相当的性能，并在端侧设备上实现高效部署。

2026-05-20cs.CLcs.LG

Semi-tied Units for Efficient Gating in LSTM and Highway Networks

Cambridge University（剑桥大学）

Gating is a key technique used for integrating information from multiple sources by long short-term memory (LSTM) models and has recently also been applied to other models such as the highway network. Although gating is powerful, it is rather expensive in terms of both computation and storage as each gating unit uses a separate full weight matrix. This issue can be severe since several gates can be used together in e.g. an LSTM cell. This paper proposes a semi-tied unit (STU) approach to solve this efficiency issue, which uses one shared weight matrix to replace those in all the units in the s…

2018-06-18cs.CLcs.LGeess.AS

Nonlinear Tensor Ring Network

City University of Hong Kong（香港城市大学）；National University of Singapore（新加坡国立大学）

The state-of-the-art deep neural networks (DNNs) have been widely applied for various real-world applications, and achieved significant performance for cognitive problems. However, the increment of DNNs' width and depth in architecture results in a huge amount of parameters to challenge the storage and memory cost, limiting to the usage of DNNs on resource-constrained platforms, such as portable devices. By converting redundant models into compact ones, compression technique appears to be a practical solution to reducing the storage and memory consumption. In this paper, we develop a nonlinear…

2021-11-12cs.AIcs.LGeess.IV

MC#：面向混合专家大模型的混合压缩器

MC#: Mixture Compressor for Mixture-of-Experts Large Models

Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；NVIDIA Research（NVIDIA研究）

提出MC#框架，结合静态混合精度量化（PMQ）与动态专家剪枝（OTP），在DeepSeek-VL2上实现6.2倍权重压缩且精度仅降1.7%，有效降低MoE大模型的存储与计算开销。

2025-10-13cs.AIcs.LG

利用副本法评估量子感知器的存储容量

Storage Capacity Evaluation of the Quantum Perceptron using the Replica Method

本文利用副本法评估基于重复直到方法实现的量子感知器的存储容量，发现其容量大于经典感知器，原因在于量子感知器虽结构简单但激活函数具有高度非线性。

2024-04-30cond-mat.dis-nn

Correlations between hidden units in multilayer neural networks and replica symmetry breaking

We consider feed-forward neural networks with one hidden layer, tree architecture and a fixed hidden-to-output Boolean function. Focusing on the saturation limit of the storage problem the influence of replica symmetry breaking on the distribution of local fields at the hidden units is investigated. These field distributions determine the probability for finding a specific activation pattern of the hidden units as well as the corresponding correlation coefficients and therefore quantify the division of labor among the hidden units. We find that although modifying the storage capacity and the d…

1999-06-14cond-mat.dis-nncond-mat.stat-mech

Memory retrieval dynamics and storage capacity of a modular network model of association cortex with featural decomposition

The primate heteromodal cortex presents an evident functional modularity at a mesoscopic level, with physiological and anatomical evidence pointing to it as likely substrate of long-term memory. In order to investigate some of its properties, a model of multimodular autoassociator is studied. Each of the many modules represents a neocortical functional ensemble of recurrently connected neurons and operates as a Hebbian autoassociator, storing a number of local features which it can recall upon cue. The global memory patterns are made of combinations of features sparsely distributed across the …

2021-12-07cond-mat.dis-nnphysics.bio-phq-bio.NC