首页/论文/1908.05982

arXiv 1908.059822019-08-19cs.LGmath.FAstat.ML

Iterative Neural Networks with Bounded Weights

Nicolaus Copernicus University（尼古拉·哥白尼大学）

Tomasz Piotrowski, Krzysztof Rykaczewski

英文摘要：

A recent analysis of a model of iterative neural network in Hilbert spaces established fundamental properties of such networks, such as existence of the fixed points sets, convergence analysis, and Lipschitz continuity. Building on these results, we show that under a single mild condition on the weights of the network, one is guaranteed to obtain a neural network converging to its unique fixed point. We provide a bound on the norm of this fixed point in terms of norms of weights and biases of the network. We also show why this model of a feed-forward neural network is not able to accomodate Hopfield networks under our assumption.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

有界更新的迭代学习算法的泛化误差界

Generalization error bounds for iterative learning algorithms with bounded updates

Xi’an Jiaotong University（西安交通大学）

本文用信息论方法为有界更新迭代学习算法提出新的泛化误差界，通过更新不确定性和方差分解改进界，并联系大语言模型缩放行为。

2023-10-14cs.LGstat.ML

Bounded Risk-Sensitive Markov Games: Forward Policy Design and Inverse Reward Learning with Iterative Reasoning and Cumulative Prospect Theory

University of California, Berkeley（加州大学伯克利分校）

Classical game-theoretic approaches for multi-agent systems in both the forward policy design problem and the inverse reward learning problem often make strong rationality assumptions: agents perfectly maximize expected utilities under uncertainties. Such assumptions, however, substantially mismatch with observed humans' behaviors such as satisficing with sub-optimal, risk-seeking, and loss-aversion decisions. In this paper, we investigate the problem of bounded risk-sensitive Markov Game (BRSMG) and its inverse reward learning problem for modeling human realistic behaviors and learning human …

2021-03-21cs.LGstat.ML

逆流倍增网络：一种受肾脏启发的具有可证明有界不动点动力学的迭代算子

Countercurrent Multiplier Networks: A Renal-Inspired Iterative Operator with Provably Bounded Fixed-Point Dynamics

研究受肾脏逆流倍增机制启发的可微序列算子CCM层，作为残差迭代细化的替代方案，用于实现尿液浓缩类似效果，有望为神经架构带来新方法。

2026-07-22cs.LGmath-phmath.MP

Retraining-Based Iterative Weight Quantization for Deep Neural Networks

Samsung Research（三星研究院）

Model compression has gained a lot of attention due to its ability to reduce hardware resource requirements significantly while maintaining accuracy of DNNs. Model compression is especially useful for memory-intensive recurrent neural networks because smaller memory footprint is crucial not only for reducing storage requirement but also for fast inference operations. Quantization is known to be an effective model compression method and researchers are interested in minimizing the number of bits to represent parameters. In this work, we introduce an iterative technique to apply quantization, pr…

2018-05-29cs.LGstat.ML

Iterative temporal differencing with random synaptic feedback weights support error backpropagation for deep learning

University of Rhode Island（罗德岛大学）

This work shows that a differentiable activation function is not necessary any more for error backpropagation. The derivative of the activation function can be replaced by an iterative temporal differencing using fixed random feedback alignment. Using fixed random synaptic feedback alignment with an iterative temporal differencing is transforming the traditional error backpropagation into a more biologically plausible approach for learning deep neural network architectures. This can be a big step toward the integration of STDP-based error backpropagation in deep learning.

2019-07-15cs.LGcs.NE

Randomly Initialized Subnetworks with Iterative Weight Recycling

Colorado State University（科罗拉多州立大学）

The Multi-Prize Lottery Ticket Hypothesis posits that randomly initialized neural networks contain several subnetworks that achieve comparable accuracy to fully trained models of the same architecture. However, current methods require that the network is sufficiently overparameterized. In this work, we propose a modification to two state-of-the-art algorithms (Edge-Popup and Biprop) that finds high-accuracy subnetworks with no additional storage cost or scaling. The algorithm, Iterative Weight Recycling, identifies subsets of important weights within a randomly initialized network for intra-la…

2023-03-28cs.AIcs.LG

更多相关论文收起更多论文

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

Rakuten Mobile, Inc.（乐天移动公司）

In low-latency or mobile applications, lower computation complexity, lower memory footprint and better energy efficiency are desired. Many prior works address this need by removing redundant parameters. Parameter quantization replaces floating-point arithmetic with lower precision fixed-point arithmetic, further reducing complexity. Typical training of quantized weight neural networks starts from fully quantized weights. Quantization creates random noise. As a way to compensate for this noise, during training, we propose to quantize some weights while keeping others in floating-point precision…

2021-11-13cs.LG

Pruning Randomly Initialized Neural Networks with Iterative Randomization

NTT Computer and Data Science Laboratories, NTT Corporation（NTT公司NTT计算机与数据科学实验室）；NTT Social Informatics Laboratories, NTT Corporation（NTT公司NTT社会信息学实验室）

Pruning the weights of randomly initialized neural networks plays an important role in the context of lottery ticket hypothesis. Ramanujan et al. (2020) empirically showed that only pruning the weights can achieve remarkable performance instead of optimizing the weight values. However, to achieve the same level of performance as the weight optimization, the pruning approach requires more parameters in the networks before pruning and thus more memory space. To overcome this parameter inefficiency, we introduce a novel framework to prune randomly initialized neural networks with iteratively rand…

2022-04-05cs.AIcs.LGcs.NE

用于迭代幅度剪枝的多粒子稀疏权重平均

Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning

KAIST AI（韩国科学技术院人工智能学院）；AITRICS（AITRICS公司）

针对大规模神经网络的剪枝问题，提出多粒子稀疏权重平均（SWAMP）方法，通过并行训练多个稀疏模型并权重平均生成单一掩码，在不同稀疏度下持续优于现有基线。

2024-04-26cs.AIcs.LG

BRAID：通过权重绑定迭代图神经网络学习相互依存安全博弈中的均衡映射

BRAID: Learning Equilibrium Maps in Interdependent Security Games via Weight-Tied Iterative Graph Neural Networks

The Ohio State University（俄亥俄州立大学）

针对网络上相互依存安全博弈中纳什均衡计算成本高的问题，提出BRAID模型，用权重绑定迭代图神经网络学习均衡映射，速度提升43倍，可预测均衡并恢复参数扰动下的均衡变化，经实验验证有效。

2026-08-18cs.GTcs.LG

WeightMom: Learning Sparse Networks using Iterative Momentum-based pruning

University of Washington（华盛顿大学）；New York University（纽约大学）；Unipart Group（优尼帕特集团）

Deep Neural Networks have been used in a wide variety of applications with significant success. However, their highly complex nature owing to comprising millions of parameters has lead to problems during deployment in pipelines with low latency requirements. As a result, it is more desirable to obtain lightweight neural networks which have the same performance during inference time. In this work, we propose a weight based pruning approach in which the weights are pruned gradually based on their momentum of the previous iterations. Each layer of the neural network is assigned an importance valu…

2022-08-11cs.CVcs.LG