Neural networks are the state-of-the-art approach for many tasks and the activation function is one of the main building blocks that allow such performance. Recently, a novel transformative adaptive activation function (TAAF) allowing for any vertical and horizontal translation and scaling was proposed. This work sets the TAAF into the context of other activation functions. It shows that the TAAFs generalize over 50 existing activation functions and utilize similar concepts as over 70 other activation functions, underscoring the versatility of TAAFs. This comprehensive exploration positions TAAFs as a promising and adaptable addition to neural networks.
In this study, we investigate the generalization of LSTM, ReLU and GRU models on counting tasks over long sequences. Previous theoretical work has established that RNNs with ReLU activation and LSTMs have the capacity for counting with suitable configuration, while GRUs have limitations that prevent correct counting over longer sequences. Despite this and some positive empirical results for LSTMs on Dyck-1 languages, our experimental results show that LSTMs fail to learn correct counting behavior for sequences that are significantly longer than in the training data. ReLUs show much larger vari…
Vienna University of Technology（维也纳工业大学）；Politecnico di Torino（都灵理工大学）
Activation functions influence behavior and performance of DNNs. Nonlinear activation functions, like Rectified Linear Units (ReLU), Exponential Linear Units (ELU) and Scaled Exponential Linear Units (SELU), outperform the linear counterparts. However, selecting an appropriate activation function is a challenging problem, as it affects the accuracy and the complexity of the given DNN. In this paper, we propose a novel methodology to automatically select the best-possible activation function for each layer of a given DNN, such that the overall DNN accuracy, compared to considering only one type…
Shandong University（山东大学）；CSIRO（澳大利亚联邦科学与工业研究组织）；Nanjing University of Science and Technology（南京理工大学）；Shandong Jianzhu University（山东建筑大学）
Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs). Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs, i.e., whether there exists a part of features arriving at the linear separable feature space in the intermediate layers, that does not require…
Faculty of Engineering, North-West University（西北大学工程学院）；North-West University（西北大学）
While deep neural networks (DNNs) have become a standard architecture for many machine learning tasks, their internal decision-making process and general interpretability is still poorly understood. Conversely, common decision trees are easily interpretable and theoretically well understood. We show that by encoding the discrete sample activation values of nodes as a binary representation, we are able to extract a decision tree explaining the classification procedure of each layer in a ReLU-activated multilayer perceptron (MLP). We then combine these decision trees with existing feature attrib…
Exploring Winograd Convolution for Cost-effective Neural Network Fault Tolerance
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Control Engineering（北京控制工程研究所）；Institute of High Performance Computing, A*STAR（新加坡科技研究局高性能计算研究所）
In this paper, we investigate the relationship between diversity metrics, accuracy, and resiliency to natural image corruptions of Deep Learning (DL) image classifier ensembles. We investigate the potential of an attribution-based diversity metric to improve the known accuracy-diversity trade-off of the typical prediction-based diversity. Our motivation is based on analytical studies of design diversity that have shown that a reduction of common failure modes is possible if diversity of design choices is achieved. Using ResNet50 as a comparison baseline, we evaluate the resiliency of multiple …
Preferred Networks（优选网络公司）；The University of Tokyo（东京大学）
Tensor decomposition methods are widely used for model compression and fast inference in convolutional neural networks (CNNs). Although many decompositions are conceivable, only CP decomposition and a few others have been applied in practice, and no extensive comparisons have been made between available methods. Previous studies have not determined how many decompositions are available, nor which of them is optimal. In this study, we first characterize a decomposition class specific to CNNs by adopting a flexible graphical notation. The class includes such well-known CNN modules as depthwise s…
University of Missouri（密苏里大学）；The University of Texas at Dallas（德克萨斯大学达拉斯分校）
Systolic array-based deep neural network (DNN) accelerators have recently gained prominence for their low computational cost. However, their high energy consumption poses a bottleneck to their deployment in energy-constrained devices. To address this problem, approximate computing can be employed at the cost of some tolerable accuracy loss. However, such small accuracy variations may increase the sensitivity of DNNs towards undesired subtle disturbances, such as permanent faults. The impact of permanent faults in accurate DNNs has been thoroughly investigated in the literature. Conversely, the…
City University of Hong Kong（香港城市大学）；Baidu USA LLC（百度美国有限责任公司）
Model pruning aims to reduce the deep neural network (DNN) model size or computational overhead. Traditional model pruning methods such as l-1 pruning that evaluates the channel significance for DNN pay too much attention to the local analysis of each channel and make use of the magnitude of the entire feature while ignoring its relevance to the batch normalization (BN) and ReLU layer after each convolutional operation. To overcome these problems, we propose a new model pruning method from a new perspective of gradient flow in this paper. Specifically, we first theoretically analyze the channe…
Structured Progressive Knowledge Activation for LLM-Driven Neural Architecture Search
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University.（人机混合增强智能国家重点实验室，人工智能与机器人研究院，西安交通大学）；MiLM Plus, Xiaomi Inc.（小米MiLM Plus）；North China University of Technology, Beijing.（华北电力大学（北京））；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）
Exploring Geographic Relative Space in Large Language Models through Activation Patching
School of Geography, Geology and the Environment, University of Leicester, UK（地理、地质与环境学院，莱斯特大学，英国）；Department of Mathematics, Computer Science and Physics, University of Udine, Italy（数学、计算机科学与物理系，乌迪内大学，意大利）
Exploring the Promise and Limits of Real-Time Recurrent Learning
Center for Brain Science, Harvard University（哈佛大学脑科学中心）；The Swiss AI Lab, IDSIA, USI & SUPSI（瑞士IDSIA人工智能实验室（隶属于瑞士意大利语区大学和瑞士应用科学大学））；AI Initiative, KAUST（阿卜杜拉国王科技大学人工智能倡议）
As deep neural networks are increasingly used in solving high-stake problems, there is a pressing need to understand their internal decision mechanisms. Visualization has helped address this problem by assisting with interpreting complex deep neural networks. However, current tools often support only single data instances, or visualize layers in isolation. We present NeuralDivergence, an interactive visualization system that uses activation distributions as a high-level summary of what a model has learned. NeuralDivergence enables users to interactively summarize and compare activation distrib…
Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost, and thus it has become a trend in model scaling. Still it is a mystery how MoE layers bring quality gains by leveraging the parameters with sparse activation. In this work, we investigate several key factors in sparse expert models. We observe that load imbalance may not be a significant problem affecting model quality, contrary to the perspectives of recent studies, while the number of sparsely activated experts $k$ and expert capacity $C$ in top-$k$ routing …
Deep neural networks such as GoogLeNet, ResNet, and BERT have achieved impressive performance in tasks such as image and text classification. To understand how such performance is achieved, we probe a trained deep neural network by studying neuron activations, i.e., combinations of neuron firings, at various layers of the network in response to a particular input. With a large number of inputs, we aim to obtain a global view of what neurons detect by studying their activations. In particular, we develop visualizations that show the shape of the activation space, the organizational principle be…
Department of Electrical Engineering, Tsinghua University（清华大学电子工程系）
The developments of deep neural networks (DNN) in recent years have ushered a brand new era of artificial intelligence. DNNs are proved to be excellent in solving very complex problems, e.g., visual recognition and text understanding, to the extent of competing with or even surpassing people. Despite inspiring and encouraging success of DNNs, thorough theoretical analyses still lack to unravel the mystery of their magics. The design of DNN structure is dominated by empirical results in terms of network depth, number of neurons and activations. A few of remarkable works published recently in an…
Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes
Institute of Computer Science（计算机科学研究所）；Ruhr West University of Applied Science（鲁尔西应用科学大学）；Ruhr University Bochum（波鸿鲁尔大学）；Institute for Neural Computation（神经计算研究所）
Asia University（亚洲大学）；China Medical University Hospital, China Medical University（中国医药大学附设医院，中国医药大学）；Universitas Muhammadiyah Yogyakarta（穆罕默迪亚日惹大学）；Western Norway University of Applied Sciences（挪威西部应用科学大学）
The spread of PM2.5 pollutants that endanger health is difficult to predict because it involves many atmospheric variables. These micron particles can spread rapidly from their source to residential areas, increasing the risk of respiratory disease if exposed for long periods. The prediction system of PM2.5 propagation provides more detailed and accurate information as an early warning system to reduce health impacts on the community. According to the idea of transformative computing, the approach we propose in this paper allows computation on the dataset obtained from massive-scale PM2.5 sens…
Exploring Unsupervised Cell Recognition with Prior Self-activation Maps
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Research Center for Industries of the Future, Westlake University（西湖大学未来产业研究中心）；School of Engineering, Westlake University（西湖大学工学院）；Westlake University（西湖大学）；Zhejiang University（浙江大学）
University of Tokyo（东京大学）；University of Cambridge（剑桥大学）；National Chiao Tung University（国立交通大学）；Japan Science and Technology Agency（日本科学技术振兴机构）
In this paper, we present contemporary techniques for visualising the feature space of a deep learning image classification neural network. These techniques are viewed in the context of a feed-forward network trained to classify low resolution fluorescence images of white blood cells captured using optofluidic imaging. The model has two output classes corresponding to two different cell types, which are often difficult to distinguish by eye. This paper has two major sections. The first looks to develop the information space presented by dimension reduction techniques, such as t-SNE, used to em…
Exploring specialization and sensitivity of convolutional neural networks in the context of simultaneous image augmentations
Marchuk Institute of Numerical Mathematics（马尔库克数值数学研究所）；Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究所）；Faculty of Computational Mathematics and Cybernetics, Lomonosov Moscow State University（洛蒙诺索夫莫斯科国立大学计算数学与 cybernetics 学院）；Artificial Intelligence Research Institute (AIRI)（人工智能研究 institute (AIRI)）
DORA: Exploring Outlier Representations in Deep Neural Networks
Leibniz Institute for Agriculture and Bioeconomy (ATB)（莱布尼茨农业与生物经济研究所（ATB））；Technical University of Berlin（柏林工业大学）；BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所（BIFOLD））；Korea University（高丽大学）；Max Planck Institut für Informatik（马克斯·普朗克信息学研究所）；Google Research（谷歌研究院）；University of Potsdam（波茨坦大学）；UiT Arctic University of Norway（挪威北极大学）
Model compression becomes a recent trend due to the requirement of deploying neural networks on embedded and mobile devices. Hence, both accuracy and efficiency are of critical importance. To explore a balance between them, a knowledge distillation strategy is proposed for general visual representation learning. It utilizes our well-designed activation map adaptive module to replace some blocks of the teacher network, exploring the most appropriate supervisory features adaptively during the training process. Using the teacher's hidden layer output to prompt the student network to train so as t…
Institute of Computing Technology（计算技术研究所）；Chinese Academy of Science（中国科学院）；Peking University（北京大学）；Fudan University（复旦大学）
In this work, we propose a generalized likelihood ratio method capable of training the artificial neural networks with some biological brain-like mechanisms,.e.g., (a) learning by the loss value, (b) learning via neurons with discontinuous activation and loss functions. The traditional back propagation method cannot train the artificial neural networks with aforementioned brain-like learning mechanisms. Numerical results show that the robustness of various artificial neural networks trained by the new method is significantly improved when the input data is affected by both the natural noises a…
Inception Institute of Artificial Intelligence（起源人工智能研究院）
Conditioning analysis uncovers the landscape of an optimization objective by exploring the spectrum of its curvature matrix. This has been well explored theoretically for linear models. We extend this analysis to deep neural networks (DNNs) in order to investigate their learning dynamics. To this end, we propose layer-wise conditioning analysis, which explores the optimization landscape with respect to each layer independently. Such an analysis is theoretically supported under mild assumptions that approximately hold in practice. Based on our analysis, we show that batch normalization (BN) can…
Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Exploration of Activation Fault Reliability in Quantized Systolic Array-Based DNN Accelerators
Tallinn University of Technology（塔林理工大学）；University of Alberta（阿尔伯塔大学）；Shahid Bahonar University of Kerman（克尔曼沙希德·巴霍纳尔大学）；York University（约克大学）；Mälardalen University（梅拉达伦大学）
Quantization is an essential step in the efficient deployment of deep learning models and as such is an increasingly popular research topic. An important practical aspect that is not addressed in the current literature is how to analyze and fix fail cases where the use of quantization results in excessive degradation. In this paper, we present a simple analytic framework that breaks down overall degradation to its per layer contributions. We analyze many common networks and observe that a layer's contribution is determined by both intrinsic (local) factors - the distribution of the layer's wei…
