Understanding the role of (stochastic) gradient descent (SGD) in the training and generalisation of deep neural networks (DNNs) with ReLU activation has been the object study in the recent past. In this paper, we make use of deep gated networks (DGNs) as a framework to obtain insights about DNNs with ReLU activation. In DGNs, a single neuronal unit has two components namely the pre-activation input (equal to the inner product the weights of the layer and the previous layer outputs), and a gating value which belongs to $[0,1]$ and the output of the neuronal unit is equal to the multiplication of pre-activation input and the gating value. The standard DNN with ReLU activation, is a special case of the DGNs, wherein the gating value is $1/0$ based on whether or not the pre-activation input is positive or negative. We theoretically analyse and experiment with several variants of DGNs, each variant suited to understand a particular aspect of either training or generalisation in DNNs with ReLU activation. Our theory throws light on two questions namely i) why increasing depth till a point helps in training and ii) why increasing depth beyond a point hurts training? We also present experimental evidence to show that gate adaptation, i.e., the change of gating value through the course of training is key for generalisation.
Columbia University（哥伦比亚大学）；Flatiron Institute（弗莱托恩研究所）
Understanding the asymptotic behavior of gradient-descent training of deep neural networks is essential for revealing inductive biases and improving network performance. We derive the infinite-time training limit of a mathematically tractable class of deep nonlinear neural networks, gated linear networks (GLNs), and generalize these results to gated networks described by general homogeneous polynomials. We study the implications of our results, focusing first on two-layer GLNs. We then apply our theoretical predictions to GLNs trained on MNIST and show how architectural constraints and the imp…
Contemporary neural networks have achieved a series of developments and successes in many aspects; however, when exposed to data outside the training distribution, they may fail to predict correct answers. In this work, we were concerned about this generalisation issue and thus analysed a broad set of models systematically and robustly over long stories. Related experiments were conducted based on the CLUTRR, which is a diagnostic benchmark suite that can analyse generalisation of natural language understanding (NLU) systems by training over small story graphs and testing on larger ones. In or…
Central South University（中南大学）；Tianjin University（天津大学）
The online programing services, such as Github,TopCoder, and EduCoder, have promoted a lot of social interactions among the service users. However, the existing social interactions is rather limited and inefficient due to the rapid increasing of source-code repositories, which is difficult to explore manually. The emergence of source-code mining provides a promising way to analyze those source codes, so that those source codes can be relatively easy to understand and share among those service users. Among all the source-code mining attempts,program classification lays a foundation for various …
Xiamen University（厦门大学）；Fudan University（复旦大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Fujian University of Technology（福建理工大学）
In human-computer interaction, Speech Emotion Recognition (SER) plays an essential role in understanding the user's intent and improving the interactive experience. While similar sentimental speeches own diverse speaker characteristics but share common antecedents and consequences, an essential challenge for SER is how to produce robust and discriminative representations through causality between speech emotions. In this paper, we propose a Gated Multi-scale Temporal Convolutional Network (GM-TCNet) to construct a novel emotional causality representation learning component with a multi-scale r…
Beihang University（北京航空航天大学）；University of Ottawa（渥太华大学）
In this paper, we propose a test, called Flagged-1-Bit (F1B) test, to study the intrinsic capability of recurrent neural networks in sequence learning. Four different recurrent network models are studied both analytically and experimentally using this test. Our results suggest that in general there exists a conflict between feature selection and feature memorization in sequence learning. Such a conflict can be resolved either using a gating mechanism as in LSTM, or by increasing the state dimension as in Vanilla RNN. Gated models resolve this conflict by adaptively adjusting their state-update…
EECS Queen Mary University of London（伦敦玛丽女王大学电子工程与计算机科学学院）；Imperial Business School Imperial College London（帝国理工学院帝国理工商学院）
We propose a neural network approach to price EU call options that significantly outperforms some existing pricing models and comes with guarantees that its predictions are economically reasonable. To achieve this, we introduce a class of gated neural networks that automatically learn to divide-and-conquer the problem space for robust and accurate pricing. We then derive instantiations of these networks that are 'rational by design' in terms of naturally encoding a valid call option surface that enforces no arbitrage principles. This integration of human insight within data-driven learning pro…
Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through its continuous time version, namely stochastic gradient flow. We explicitly characterise the solution chosen by the stochastic flow and prove that it always enjoys better generalisation properties than that of gradient flow. Quite surprisingly, we show that the convergence speed of the training loss controls the magnitude of the biasing effect…
The introduction of Transformer model has led to tremendous advancements in sequence modeling, especially in text domain. However, the use of attention-based models for video understanding is still relatively unexplored. In this paper, we introduce Gated Adversarial Transformer (GAT) to enhance the applicability of attention-based models to videos. GAT uses a multi-level attention gate to model the relevance of a frame based on local and global contexts. This enables the model to understand the video at various granularities. Further, GAT uses adversarial training to improve model generalizati…
Sainsbury Wellcome Centre, UCL（伦敦大学学院塞恩斯伯里·韦尔科姆中心）；Meta AI；CIFAR（加拿大高等研究院）
Our theoretical understanding of deep learning has not kept pace with its empirical success. While network architecture is known to be critical, we do not yet understand its effect on learned representations and network behavior, or how this architecture should reflect task structure.In this work, we begin to address this gap by introducing the Gated Deep Linear Network framework that schematizes how pathways of information flow impact learning dynamics within an architecture. Crucially, because of the gating, these networks can compute nonlinear functions of their input. We derive an exact re…
Gated Domain Units for Multi-source Domain Generalization
ETH Zurich（苏黎世联邦理工学院）；CISPA - Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Karlsruhe Institute for Technology（卡尔斯鲁厄理工学院）；University of St. Gallen（圣加仑大学）
Recently recurrent neural networks (RNN) has been very successful in handling sequence data. However, understanding RNN and finding the best practices for RNN is a difficult task, partly because there are many competing and complex hidden units (such as LSTM and GRU). We propose a gated unit for RNN, named as Minimal Gated Unit (MGU), since it only contains one gate, which is a minimal design among all gated hidden units. The design of MGU benefits from evaluation results on LSTM and GRU in the literature. Experiments on various sequence data show that MGU has comparable accuracy with GRU, but…
On progressive sharpening, flat minima and generalisation
Mathematical Institute for Data Science（数据科学数学研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）
Imperial College London（伦敦帝国学院）；Google Research（谷歌研究院）
State-of-the-art pre-trained language models have been shown to memorise facts and perform well with limited amounts of training data. To gain a better understanding of how these models learn, we study their generalisation and memorisation capabilities in noisy and low-resource scenarios. We find that the training of these models is almost unaffected by label noise and that it is possible to reach near-optimal results even on extremely noisy datasets. However, our experiments also show that they mainly learn from high-frequency patterns and largely fail when tested on low-resource tasks such a…
We present several methods to improve the generalisation of language identification (LID) systems to new speakers and to new domains. These methods involve Spectral augmentation, where spectrograms are masked in the frequency or time bands during training and CNN architectures that are pre-trained on the Imagenet dataset. The paper also introduces the novel Triplet Entropy Loss training method, which involves training a network simultaneously using Cross Entropy and Triplet loss. It was found that all three methods improved the generalisation of the models, though not significantly. Even thoug…
Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；The University of Queensland（昆士兰大学）；Univ. Grenoble Alpes（格勒诺布尔大学）；Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆廷诺实验室）
Shahjalal University of Science and Technology（沙赫贾拉尔科技大学）；University of Alberta（阿尔伯塔大学）；Fordham University（福特汉姆大学）
Forensic analysis of manipulated pixels requires the identification of various hidden and subtle features from images. Conventional image recognition models generally fail at this task because they are biased and more attentive toward the dominant local and spatial features. In this paper, we propose a novel Gated Context Attention Network (GCA-Net) that utilizes non-local attention in conjunction with a gating mechanism in order to capture the finer image discrepancies and better identify forged regions. The proposed framework uses high dimensional embeddings to filter and aggregate the relev…
Event Extraction plays an important role in information-extraction to understand the world. Event extraction could be split into two subtasks: one is event trigger extraction, the other is event arguments extraction. However, the F-Score of event arguments extraction is much lower than that of event trigger extraction, i.e. in the most recent work, event trigger extraction achieves 80.7%, while event arguments extraction achieves only 58%. In pipelined structures, the difficulty of event arguments extraction lies in its lack of classification feature, and the much higher computation consumptio…
American International University-Bangladesh（美国国际大学-孟加拉国分校）
There is very little notable research on generating descriptions of the Bengali language. About 243 million people speak in Bengali, and it is the 7th most spoken language on the planet. The purpose of this research is to propose a CNN and Bidirectional GRU based architecture model that generates natural language captions in the Bengali language from an image. Bengali people can use this research to break the language barrier and better understand each other's perspectives. It will also help many blind people with their everyday lives. This paper used an encoder-decoder approach to generate ca…
Deep Neural Networks as the Semi-classical Limit of Topological Quantum Neural Networks: The problem of generalisation
Laboratori Nazionali di Frascati INFN（意大利国家核物理研究院弗拉斯卡蒂国家实验室）；INFN sezione Roma “Tor Vergata”（意大利国家核物理研究院罗马第二大学分部）；Department of Mathematics and Statistics, Idaho State University（爱达荷州立大学数学与统计系）；Department of Physics, University of Rome “La Sapienza”（罗马第一大学物理系）；Department of Mechanics and Aerospace Engineering, Southern University of Science and Technology（南方科技大学力学与航空航天工程系）；Allen Discovery Center, Tufts University（塔夫茨大学艾伦发现中心）；College of Design and Innovation, Tongji University（同济大学设计与创新学院）
Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures
University of Science - VNUHCM（越南胡志明市科学大学）；ARC Centre of Excellence for the Mathematical Analysis of Cellular System（细胞系统数学分析卓越研究中心）；School of Mathematical Sciences Queensland University of Technology（昆士兰理工大学数学科学学院）；Department of Statistics University of Michigan（密歇根大学统计学系）；Department of Statistics and Data Science University of Texas at Austin（德克萨斯大学奥斯汀分校统计学与数据科学系）
Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisation
McGovern Institute for Brain Research, Massachusetts Institute of Technology, MA 02139, U.S.A.（麦戈文脑科学研究所，麻省理工学院）；K. Lisa Yang Integrative Computational Neuroscience Center in the Yang-Tan Collective（李嘉诚整合计算神经科学中心）；Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, MA 02139, U.S.A.（脑科学与认知科学系，麻省理工学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, MA 02139, U.S.A.（电气工程与计算机科学系，麻省理工学院）
