Our objective is to estimate the unknown compositional input from its output response through an unknown system after estimating the inverse of the original system with a training set. The proposed methods using artificial neural networks (ANNs) can compete with the optimal bounds for linear systems, where convex optimization theory applies, and demonstrate promising results for nonlinear system inversions. We performed extensive experiments by designing numerous different types of nonlinear systems.
Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam -- a multiplicative version of the Adam optimiser -- and show that it can train state of the art neural network architectures without learning rate tuning.…
Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations. Traditional methods first devise task-specific operations such as Synonym Substitute, then preset the corresponding parameters such as the substitution rate artificially, which require a lot of prior knowledge and are prone to fall into the sub-optimum. Besides, the number of editing operations is limited in the previous methods, which decreases the diversity of the augmented data and thus restricts the performance gain. To overcome the above limitations, we p…
Methods of deep learning have become increasingly popular in recent years, but they have not arrived in compositional data analysis. Imputation methods for compositional data are typically applied on additive, centered or isometric log-ratio representations of the data. Generally, methods for compositional data analysis can only be applied to observed positive entries in a data matrix. Therefore one tries to impute missing values or measurements that were below a detection limit. In this paper, a new method for imputing rounded zeros based on artificial neural networks is shown and compared wi…
Shandong University（山东大学）；University of Geneva（日内瓦大学）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters pre…
German Aerospace Center e.V (DLR)（德国航空航天中心（DLR））；Carl von Ossietzky University Oldenburg（奥尔登堡卡尔·冯·奥西茨基大学）
This paper presents an approach for verifying the behaviour of nonlinear Artificial Neural Networks (ANNs) found in cyber-physical safety-critical systems. We implement a dedicated interval constraint propagator for the sigmoid function into the SMT solver iSAT and compare this approach with a compositional approach encoding the sigmoid function by basic arithmetic features available in iSAT and an approximating approach. Our experimental results show that the dedicated and the compositional approach clearly outperform the approximating approach. Throughout all our benchmarks, the dedicated ap…
Effectively modeling phenomena present in highly nonlinear dynamical systems whilst also accurately quantifying uncertainty is a challenging task, which often requires problem-specific techniques. We present a novel, domain-agnostic approach to tackling this problem, using compositions of physics-informed random features, derived from ordinary differential equations. The architecture of our model leverages recent advances in approximate inference for deep Gaussian processes, such as layer-wise weight-space approximations which allow us to incorporate random Fourier features, and stochastic var…
Explainable Artificial Intelligence Model for Evaluating Shear Strength Parameters of Municipal Solid Waste Across Diverse Compositional Profiles
The Joint Graduate School of Energy and Environment, King Mongkut's University of Technology Thonburi（泰国国王科技大学吞武里校区能源与环境联合研究生院）；AI Research Group, Department of Civil Engineering, Faculty of Engineering, King Mongkut's University of Technology Thonburi（泰国国王科技大学吞武里校区工程学院土木工程系人工智能研究组）；Center of Excellence on Energy Technology and Environment (CEE), PERDO, Ministry of Higher Education, Science, Research and Innovation (MHESI)（高等教育、科学、研究与创新部PERDO能源技术与环境卓越中心）；Department of Environmental Engineering, Faculty of Engineering, Kasetsart University（农业大学工程学院环境工程系）
University of Edinburgh（爱丁堡大学）；University of Osnabrück（奥斯纳布吕克大学）；Facebook AI Research（脸书人工智能研究院）
Obtaining human-like performance in NLP is often argued to require compositional generalisation. Whether neural networks exhibit this ability is usually studied by training models on highly compositional synthetic data. However, compositionality in natural language is much more complex than the rigid, arithmetic-like version such data adheres to, and artificial compositionality tests thus do not allow us to determine how neural models deal with more realistic forms of compositionality. In this work, we re-instantiate three compositionality tests from the literature and reformulate them for neu…
Finding an embedding space for a linear approximation of a nonlinear dynamical system enables efficient system identification and control synthesis. The Koopman operator theory lays the foundation for identifying the nonlinear-to-linear coordinate transformations with data-driven methods. Recently, researchers have proposed to use deep neural networks as a more expressive class of basis functions for calculating the Koopman operators. These approaches, however, assume a fixed dimensional state space; they are therefore not applicable to scenarios with a variable number of objects. In this pape…
Can neural nets learn logic? We approach this classic question with current methods, and demonstrate that recurrent neural networks can learn to recognize first order logical entailment relations between expressions. We define an artificial language in first-order predicate logic, generate a large dataset of sample 'sentences', and use an automatic theorem prover to infer the relation between random pairs of such sentences. We describe a Siamese neural architecture trained to predict the logical relation, and experiment with recurrent and recursive networks. Siamese Recurrent Networks are surp…
Many aspects of human reasoning, including language, require learning rules from very little data. Humans can do this, often learning systematic rules from very few examples, and combining these rules to form compositional rule-based systems. Current neural architectures, on the other hand, often fail to generalize in a compositional manner, especially when evaluated in ways that vary systematically from training. In this work, we present a neuro-symbolic model which learns entire rule systems from a small set of examples. Instead of directly predicting outputs from inputs, we train our model …
Google Research（谷歌研究院）；Istituto Dalle Molle di studi sull’intelligenza artificiale (IDSIA)（达莱莫莱人工智能研究所（IDSIA））；Università della Svizzera Italiana (USI)（瑞士意大利语区大学（USI））；Scuola universitaria professionale della Svizzera italiana (SUPSI)（瑞士意大利语区应用科学大学（SUPSI））
Contemporary neural networks still fall short of human-level generalization, which extends far beyond our direct experiences. In this paper, we argue that the underlying cause for this shortcoming is their inability to dynamically and flexibly bind information that is distributed throughout the network. This binding problem affects their capacity to acquire a compositional understanding of the world in terms of symbol-like entities (like objects), which is crucial for generalizing in predictable and systematic ways. To address this issue, we propose a unifying framework that revolves around fo…
Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a composit…
A longstanding question in cognitive science concerns the learning mechanisms underlying compositionality in human cognition. Humans can infer the structured relationships (e.g., grammatical rules) implicit in their sensory observations (e.g., auditory speech), and use this knowledge to guide the composition of simpler meanings into complex wholes. Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is po…
Trustless Federated Learning at Edge-Scale: A Compositional Architecture for Decentralized, Verifiable, and Incentive-Aligned Coordination
Marshall School of Business University of Southern California（南加州大学马歇尔商学院）；School of Natural and Computing Sciences University of Aberdeen（阿伯丁大学自然与计算科学学院）
Position: A Theory of Deep Learning Must Include Compositional Sparsity
Center for Brains, Minds and Machines (CBMM), MIT, Cambridge, MA, USA（大脑、心智与机器中心（CBMM），麻省理工学院）；ETH Zurich, Zurich, Switzerland（苏黎世联邦理工学院）；University of Zurich, Zurich, Switzerland（苏黎世大学）；University of Milan, Milan, Italy（米兰大学）
Compositional Neural Operators for Multi-Dimensional Fluid Dynamics
Mines Paris - PSL University, Centre for Material Forming (CEMEF)（巴黎矿学院-PSL大学，材料成形中心（CEMEF））；Mines Paris - PSL University, Centre for Robotics (CAOR)（巴黎矿学院-PSL大学，机器人中心（CAOR））
As you read these words you are using a complex biological neural network. You have a highly interconnected set of some neurons to facilitate your reading, breathing, motion and thinking. Each of your biological neurons, a rich assembly of tissue and chemistry, has the complexity, if not the speed, of a microprocessor. Some of your neural structure was with you at birth. Other parts have been established by experience.
Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisation
McGovern Institute for Brain Research, Massachusetts Institute of Technology, MA 02139, U.S.A.（麦戈文脑科学研究所，麻省理工学院）；K. Lisa Yang Integrative Computational Neuroscience Center in the Yang-Tan Collective（李嘉诚整合计算神经科学中心）；Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, MA 02139, U.S.A.（脑科学与认知科学系，麻省理工学院）；Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, MA 02139, U.S.A.（电气工程与计算机科学系，麻省理工学院）
The identification of black-box nonlinear state-space models requires a flexible representation of the state and output equation. Artificial neural networks have proven to provide such a representation. However, as in many identification problems, a nonlinear optimization problem needs to be solved to obtain the model parameters (layer weights and biases). A well-thought initialization of these model parameters can often avoid that the nonlinear optimization algorithm converges to a poorly performing local minimum of the considered cost function. This paper introduces an improved initializatio…
Hamburg University of Technology（汉堡工业大学）；Helmholtz-Zentrum Hereon（亥姆霍兹联合会赫尔岑研究中心）
The constitutive behavior of polymeric materials is often modeled by finite linear viscoelastic (FLV) or quasi-linear viscoelastic (QLV) models. These popular models are simplifications that typically cannot accurately capture the nonlinear viscoelastic behavior of materials. For example, the success of attempts to capture strain rate-dependent behavior has been limited so far. To overcome this problem, we introduce viscoelastic Constitutive Artificial Neural Networks (vCANNs), a novel physics-informed machine learning framework for anisotropic nonlinear viscoelasticity at finite strains. vCAN…
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Large-scale cyber-physical systems require that control policies are distributed, that is, that they only rely on local real-time measurements and communication with neighboring agents. Optimal Distributed Control (ODC) problems are, however, highly intractable even in seemingly simple cases. Recent work has thus proposed training Neural Network (NN) distributed controllers. A main challenge of NN controllers is that they are not dependable during and after training, that is, the closed-loop system may be unstable, and the training may fail due to vanishing and exploding gradients. In this pap…
Tree-structured neural networks encode a particular tree geometry for a sentence in the network design. However, these models have at best only slightly outperformed simpler sequence-based models. We hypothesize that neural sequence models like LSTMs are in fact able to discover and implicitly use recursive compositional structure, at least for tasks with clear cues to that structure in the data. We demonstrate this possibility using an artificial data task for which recursive compositional structure is crucial, and find an LSTM-based sequence model can indeed learn to exploit the underlying t…
In recent years deep artificial neural networks (DNNs) have been successfully employed in numerical simulations for a multitude of computational problems including, for example, object and face recognition, natural language processing, fraud detection, computational advertisement, and numerical approximations of partial differential equations (PDEs). These numerical simulations indicate that DNNs seem to possess the fundamental flexibility to overcome the curse of dimensionality in the sense that the number of real parameters used to describe the DNN grows at most polynomially in both the reci…
We present a method for solving linear and nonlinear PDEs based on the variable projection (VarPro) framework and artificial neural networks (ANN). For linear PDEs, enforcing the boundary/initial value problem on the collocation points leads to a separable nonlinear least squares problem about the network coefficients. We reformulate this problem by the VarPro approach to eliminate the linear output-layer coefficients, leading to a reduced problem about the hidden-layer coefficients only. The reduced problem is solved first by the nonlinear least squares method to determine the hidden-layer co…
