We propose an approach to symbolic regression based on a novel variational autoencoder for generating hierarchical structures, HVAE. It combines simple atomic units with shared weights to recursively encode and decode the individual nodes in the hierarchy. Encoding is performed bottom-up and decoding top-down. We empirically show that HVAE can be trained efficiently with small corpora of mathematical expressions and can accurately encode expressions into a smooth low-dimensional latent space. The latter can be efficiently explored with various optimization methods to address the task of symbolic regression. Indeed, random search through the latent space of HVAE performs better than random search through expressions generated by manually crafted probabilistic grammars for mathematical expressions. Finally, EDHiE system for symbolic regression, which applies an evolutionary algorithm to the latent space of HVAE, reconstructs equations from a standard symbolic regression benchmark better than a state-of-the-art system based on a similar combination of deep learning and evolutionary algorithms.ž
Deep generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have recently been applied to style and domain transfer for images, and in the case of VAEs, music. GAN-based models employing several generators and some form of cycle consistency loss have been among the most successful for image domain transfer. In this paper we apply such a model to symbolic music and show the feasibility of our approach for music genre transfer. Evaluations using separate genre classifiers show that the style transfer works well. In order to improve the fidelity of th…
Mathematical reasoning recently has been shown as a hard challenge for neural systems. Abilities including expression translation, logical reasoning, and mathematics knowledge acquiring appear to be essential to overcome the challenge. This paper demonstrates that some abilities can be achieved through abductive combination with discrete systems that have been programmed with human knowledge. On a mathematical reasoning dataset, we adopt the recently proposed abductive learning framework, and propose the ABL-Sym algorithm that combines the Transformer neural models with a symbolic mathematics …
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Machine learning applications to symbolic mathematics are becoming increasingly popular, yet there lacks a centralized source of real-world symbolic expressions to be used as training data. In contrast, the field of natural language processing leverages resources like Wikipedia that provide enormous amounts of real-world textual data. Adopting the philosophy of "mathematics as language," we bridge this gap by introducing a pipeline for distilling mathematical expressions embedded in Wikipedia into symbolic encodings to be used in downstream machine learning tasks. We demonstrate that a $\texti…
A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data
AnnLab（安澜实验室）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology（北京市半导体神经网络智能感知与计算重点实验室）；Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences（中国科学院大学材料科学与光电技术学院）
Discovering physical laws with parallel symbolic enumeration
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；School of Engineering Science, University of Chinese Academy of Sciences（中国科学院大学工程科学学院）；State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences（中国科学院力学研究所非线性力学重点实验室）；Department of Computer Science and Engineering, HKUST（香港科技大学计算机科学与工程系）
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Discovering the underlying mathematical expressions describing a dataset is a core challenge for artificial intelligence. This is the problem of $\textit{symbolic regression}$. Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are underexplored. We propose a framework that leverages deep learning for symbolic regression via a simple idea: use a large model to search the space of small models. Specifically, we use a recurrent neural network to emit a distribution over tractable mathematical expressions and employ a novel …
Symbolic regression is the task of identifying a mathematical expression that best fits a provided dataset of input and output values. Due to the richness of the space of mathematical expressions, symbolic regression is generally a challenging problem. While conventional approaches based on genetic evolution algorithms have been used for decades, deep learning-based methods are relatively new and an active research area. In this work, we present SymbolicGPT, a novel transformer-based language model for symbolic regression. This model exploits the advantages of probabilistic language models lik…
China University of Petroleum-Beijing（中国石油大学（北京））；University of Wisconsin Milwaukee（威斯康星大学密尔沃基分校）
Many recent studies focus on developing mechanisms to explain the black-box behaviors of neural networks (NNs). However, little work has been done to extract the potential hidden semantics (mathematical representation) of a neural network. A succinct and explicit mathematical representation of a NN model could improve the understanding and interpretation of its behaviors. To address this need, we propose a novel symbolic regression method for neural works (called SRNet) to discover the mathematical expressions of a NN. SRNet creates a Cartesian genetic programming (NNCGP) to represent the hidd…
Centro de investigación CITIC（CITIC研究中心）；University of A Coruña（拉科鲁尼亚大学）
This paper describes a new method for Symbolic Regression that allows to find mathematical expressions from a dataset. This method has a strong mathematical basis. As opposed to other methods such as Genetic Programming, this method is deterministic, and does not involve the creation of a population of initial solutions. Instead of it, a simple expression is being grown until it fits the data. The experiments performed show that the results are as good as other Machine Learning methods, in a very low computational time. Another advantage of this technique is that the complexity of the expressi…
Discovering Mathematical Formulas from Data via GPT-guided Monte Carlo Tree Search
ANNLAB, Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所ANNLAB）；School of Integrated Circuits, University of Chinese Academy of Sciences（中国科学院大学集成电路学院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）
LieDiscover: Adaptive Symbolic Library Construction for Explicit Open-form Symmetry Discovery
East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；Beihang University（北京航空航天大学）；Institute of Applied Physics and Computational Mathematics（应用物理与计算数学研究所）；Northeast Normal University（东北师范大学）
Integration is indispensable, not only in mathematics, but also in a wide range of other fields. A deep learning method has recently been developed and shown to be capable of integrating mathematical functions that could not previously be integrated on a computer. However, that method treats integration as equivalent to natural language translation and does not reflect mathematical information. In this study, we adjusted the learning model to take mathematical information into account and developed a wide range of learning models that learn the order of numerical operations more robustly. In t…
Northeastern University（东北大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of Notre Dame（圣母大学）；Renmin University of China（中国人民大学）
Nonlinear dynamics is ubiquitous in nature and commonly seen in various science and engineering disciplines. Distilling analytical expressions that govern nonlinear dynamics from limited data remains vital but challenging. To tackle this fundamental issue, we propose a novel Symbolic Physics Learner (SPL) machine to discover the mathematical structure of nonlinear dynamics. The key concept is to interpret mathematical operations and system state variables by computational rules and symbols, establish symbolic reasoning of mathematical formulas via expression trees, and employ a Monte Carlo tre…
We describe a neural-based method for generating exact or approximate solutions to differential equations in the form of mathematical expressions. Unlike other neural methods, our system returns symbolic expressions that can be interpreted directly. Our method uses a neural architecture for learning mathematical expressions to optimize a customizable objective, and is scalable, compact, and easily adaptable for a variety of tasks and configurations. The system has been shown to effectively find exact or approximate symbolic solutions to various differential equations with applications in natur…
Massachusetts Institute of Technology（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Identifying the mathematical relationships that best describe a dataset remains a very challenging problem in machine learning, and is known as Symbolic Regression (SR). In contrast to neural networks which are often treated as black boxes, SR attempts to gain insight into the underlying relationships between the independent variables and the target variable of a given dataset by assembling analytical functions. In this paper, we present GSR, a Generalized Symbolic Regression approach, by modifying the conventional SR optimization problem formulation, while keeping the main SR objective intact…
Scalable Neural Symbolic Regression using Control Variables
School of Computer Science（计算机科学学院）；GaTech（佐治亚理工学院）；School of Computing and Data Science（计算与数据科学学院）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；William & Mary（威廉玛丽学院）；Department of EECS（电子工程与计算机科学系）；University of Tennessee, Knoxville（田纳西大学诺克斯维尔分校）；Wake Forest University（维克森林大学）
Discovering an interpretable mathematical expression for a full wind-turbine wake with artificial intelligence enhanced symbolic regression
School of Ocean and Civil Engineering, Shanghai Jiao Tong University（上海交通大学海洋与土木工程学院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）
Adaptive Problem Generation via Symbolic Representations
Korea Advanced Institute of Science and Technology（韩国科学技术院）；Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工联盟研究技术）；Massachusetts Institute of Technology（麻省理工学院）；Singapore Management University（新加坡管理大学）
Symbolic Neural Generation with Applications to Lead Discovery in Drug Design
Dept. of Computer Science & Information Systems and APPCAIR BITS Pilani, K K Birla Goa Campus, India（计算机科学与信息系统系及APPCAIR比特纳学院，K K Birl拉果阿校区，印度）；Dept. of Computer Science & Information Systems BITS Pilani, K K Birla Goa Campus, India（计算机科学与信息系统系比特纳学院，K K Birl拉果阿校区，印度）；Department of Biochemistry, University of Cambridge, Cambridge, UK（生物化学系，剑桥大学，剑桥，英国）；School of Computer Science and Engineering University of New South Wales, Sydney（计算机科学与工程学院新南威尔士大学，悉尼）；Dr. B.R. Ambedkar Center for Biomedical Research University of Delhi, New Delhi, India（B.R.阿姆贝卡尔生物医学研究中心，德里大学，新德里，印度）；Department of Chemistry BITS Pilani, K.K. Birla Goa Campus, India（化学系比特纳学院，K.K. Birl拉果阿校区，印度）
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Many machine learning strategies designed to automate mathematical tasks leverage neural networks to search large combinatorial spaces of mathematical symbols. In contrast to traditional evolutionary approaches, using a neural network at the core of the search allows learning higher-level symbolic patterns, providing an informed direction to guide the search. When no labeled data is available, such networks can still be trained using reinforcement learning. However, we demonstrate that this approach can suffer from an early commitment phenomenon and from initialization bias, both of which limi…
Meta AI；Sorbonne Université（索邦大学）；Ecole Normale Supérieure（巴黎高等师范学院）
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this pape…
Symbolic Equation Solving via Reinforcement Learning
RIKEN Center for Emergent Matter Science (CEMS)（理化学研究所 emergen 物质科学中心（CEMS））；School of Mathematical Sciences, Queen Mary University of London（伦敦玛丽女王大学数学科学学院）；The University of Tokyo（东京大学）
Mathematical expressions were generated, evaluated and used to train neural network models based on the transformer architecture. The expressions and their targets were analyzed as a character-level sequence transduction task in which the encoder and decoder are built on attention mechanisms. Three models were trained to understand and evaluate symbolic variables and expressions in mathematics: (1) the self-attentive and feed-forward transformer without recurrence or convolution, (2) the universal transformer with recurrence, and (3) the adaptive universal transformer with recurrence and adapt…
University of Kansas（堪萨斯大学）；University of Florida（佛罗里达大学）
We demonstrate the use of symbolic regression in deriving analytical formulas, which are needed at various stages of a typical experimental analysis in collider phenomenology. As a first application, we consider kinematic variables like the stransverse mass, $M_{T2}$, which are defined algorithmically through an optimization procedure and not in terms of an analytical formula. We then train a symbolic regression and obtain the correct analytical expressions for all known special cases of $M_{T2}$ in the literature. As a second application, we reproduce the correct analytical expression for a n…
SymPlex: A Structure-Aware Transformer for Symbolic PDE Solving
Department of Mathematics, University of California, Los Angeles, Los Angeles, CA, USA（加州大学洛杉矶分校数学系）；Mechanical and Aerospace Engineering, University of California, Los Angeles, Los Angeles, CA, USA（加州大学洛杉矶分校机械与航空航天工程系）；Office of Advanced Research Computing (OARC), University of California, Los Angeles, Los Angeles, CA, USA（加州大学洛杉矶分校高级计算研究办公室）
SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Data
Nanjing University（南京大学）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
