Complexity-Aware Deep Symbolic Regression with Robust Risk-Seeking Policy Gradients
Kahlert School of Computing（卡勒特计算学院）；Scientific Computing and Imaging Institute（科学计算与成像研究所）；Department of Mechanical Engineering, University of Utah（犹他大学机械工程系）
Zachary Bastiani, Robert M. Kirby, Jacob Hochhalter, Shandian Zhe
We propose a novel deep symbolic regression approach to enhance the robustness and interpretability of data-driven mathematical expression discovery. Our work is aligned with the popular DSR framework which focuses on learning a data-specific expression generator, without relying on pretrained models or additional search or planning procedures. Despite the success of existing DSR methods, they are built on recurrent neural networks, solely guided by data fitness, and potentially meet tail barriers that can zero out the policy gradient, causing inefficient model updates. To overcome these limitations, we design a decoder-only architecture that performs attention in the frequency domain and introduce a dual-indexed position encoding to conduct layer-wise generation. Second, we propose a Bayesian information criterion (BIC)-based reward function that can automatically adjust the trade-off between expression complexity and data fitness, without the need for explicit manual tuning. Third, we develop a ranking-based weighted policy update method that eliminates the tail barriers and enhances training effectiveness. Extensive benchmarks and systematic experiments demonstrate the advantages of our approach. We have released our implementation at https://github.com/ZakBastiani/CADSR.
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Discovering the underlying mathematical expressions describing a dataset is a core challenge for artificial intelligence. This is the problem of $\textit{symbolic regression}$. Despite recent advances in training neural networks to solve complex tasks, deep learning approaches to symbolic regression are underexplored. We propose a framework that leverages deep learning for symbolic regression via a simple idea: use a large model to search the space of small models. Specifically, we use a recurrent neural network to emit a distribution over tractable mathematical expressions and employ a novel …
Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；LORIA（洛林计算机科学与应用数学实验室）；Réseau de Transport d’Electricité (Rte) R&D（法国电力传输网络研发部）
In recent years, symbolic regression has been of wide interest to provide an interpretable symbolic representation of potentially large data relationships. Initially circled to genetic algorithms, symbolic regression methods now include a variety of Deep Learning based alternatives. However, these methods still do not generalize well to real-world data, mainly because they hardly include domain knowledge nor consider physical relationships between variables such as known equations and units. Regarding these issues, we propose a Reinforcement-Based Grammar-Guided Symbolic Regression (RBG2-SR) m…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Symbolic regression is the process of identifying mathematical expressions that fit observed output from a black-box process. It is a discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic programming. In this work, we introduce a hybrid neural-guided/genetic programming approach to symbolic regression and other combinatorial optimization problems. We propose a neural-guided component used to seed the starting population of a random restart genetic programming component…
Here we present Symplectically Integrated Symbolic Regression (SISR), a novel technique for learning physical governing equations from data. SISR employs a deep symbolic regression approach, using a multi-layer LSTM-RNN with mutation to probabilistically sample Hamiltonian symbolic expressions. Using symplectic neural networks, we develop a model-agnostic approach for extracting meaningful physical priors from the data that can be imposed on-the-fly into the RNN output, limiting its search space. Hamiltonians generated by the RNN are optimized and assessed using a fourth-order symplectic integ…
Kahlert School of Computing（卡勒特计算学院）；Scientific Computing and Imaging Institute（科学计算与成像研究所）；Department of Mechanical Engineering（机械工程系）；University of Utah（犹他大学）
PruneSymNet: A Symbolic Neural Network and Pruning Algorithm for Symbolic Regression
Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology（北京半导体神经网络智能感知与计算技术重点实验室）
University of Applied Sciences Upper Austria（上奥地利应用科学大学）
The typical methods for symbolic regression produce rather abrupt changes in solution candidates. In this work, we have tried to transform symbolic regression from an optimization problem, with a landscape that is so rugged that typical analysis methods do not produce meaningful results, to one that can be compared to typical and very smooth real-valued problems. While the ruggedness might not interfere with the performance of optimization, it restricts the possibilities of analysis. Here, we have explored different aspects of a transformation and propose a simple procedure to create real-valu…
LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression
Centre for Data Science and Artificial Intelligence & School of Engineering and Computer Science, Victoria University of Wellington（惠灵顿维多利亚大学数据科学与人工智能中心及工程与计算机科学学院）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）
We consider the problem of learning free-form symbolic expressions from raw data, such as that produced by an experiment in any scientific domain. Accurate and interpretable models of scientific phenomena are the cornerstone of scientific research. Simple yet interpretable models, such as linear or logistic regression and decision trees often lack predictive accuracy. Alternatively, accurate blackbox models such as deep neural networks provide high predictive accuracy, but do not readily admit human understanding in a way that would enrich the scientific theory of the phenomenon. Many great br…
Data-Efficient Symbolic Regression via Foundation Model Distillation
Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（巴黎-罗克琴特研究所）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）；Arizona State University（亚利桑那州立大学）；Clemson University（克莱姆森大学）
ETH（苏黎世联邦理工学院）；CSEM SA（瑞士电子与微技术中心）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Symbolic equations are at the core of scientific discovery. The task of discovering the underlying equation from a set of input-output pairs is called symbolic regression. Traditionally, symbolic regression methods use hand-designed strategies that do not improve with experience. In this paper, we introduce the first symbolic regression method that leverages large scale pre-training. We procedurally generate an unbounded set of equations, and simultaneously pre-train a Transformer to predict the symbolic equation from a corresponding set of input-output-pairs. At test time, we query the model …
Symbolic-Diffusion: Deep Learning Based Symbolic Regression with D3PM Discrete Token Diffusion
Department of Mechatronics Engineering University of Waterloo（机械工程系大学水力学院）；Department of Computer Science University of Waterloo（计算机科学系大学水力学院）；Department of Statistics and Actuarial Science University of Waterloo（统计与精算科学系大学水力学院）
Symbolic regression is the task of identifying a mathematical expression that best fits a provided dataset of input and output values. Due to the richness of the space of mathematical expressions, symbolic regression is generally a challenging problem. While conventional approaches based on genetic evolution algorithms have been used for decades, deep learning-based methods are relatively new and an active research area. In this work, we present SymbolicGPT, a novel transformer-based language model for symbolic regression. This model exploits the advantages of probabilistic language models lik…
In this paper we provide a broad benchmarking of recent genetic programming approaches to symbolic regression in the context of state of the art machine learning approaches. We use a set of nearly 100 regression benchmark problems culled from open source repositories across the web. We conduct a rigorous benchmarking of four recent symbolic regression approaches as well as nine machine learning approaches from scikit-learn. The results suggest that symbolic regression performs strongly compared to state-of-the-art gradient boosting algorithms, although in terms of running times is among the sl…
Machine learning is often applied in health science to obtain predictions and new understandings of complex phenomena and relationships, but an availability of sufficient data for model training is a widespread problem. Traditional machine learning techniques, such as random forests and gradient boosting, tend to overfit when working with data sets of only a few hundred observations. This study demonstrates that for small training sets of 250 observations, symbolic regression generalises better to out-of-sample data than traditional machine learning frameworks, as measured by the coefficient o…
Ben-Gurion University（本-古里安大学）；University of Pennsylvania（宾夕法尼亚大学）
Modifying standard gradient boosting by replacing the embedded weak learner in favor of a strong(er) one, we present SyRBo: Symbolic-Regression Boosting. Experiments over 98 regression datasets show that by adding a small number of boosting stages -- between 2--5 -- to a symbolic regressor, statistically significant improvements can often be attained. We note that coding SyRBo on top of any symbolic regressor is straightforward, and the added cost is simply a few more evolutionary rounds. SyRBo is essentially a simple add-on that can be readily added to an extant symbolic regressor, often with…
Enhancing Symbolic Regression and Universal Physics-Informed Neural Networks with Dimensional Analysis
Cheriton School of Computer Science（切尔顿计算机科学学院）；Department of Electrical and Computer Engineering（电气与计算机工程系）；University of Waterloo（滑铁卢大学）；Electrical Engineering and Computer Science Department（电气工程与计算机科学系）；Department of Applied Mathematics（应用数学系）；University of Michigan（密歇根大学）
University of Utah（犹他大学）；Scientific Computing and Imaging Institute（科学计算与成像研究所）；NASA Langley Research Center（美国国家航空航天局兰利研究中心）
In this paper, we present a machine learning method for the discovery of analytic solutions to differential equations. The method utilizes an inherently interpretable algorithm, genetic programming based symbolic regression. Unlike conventional accuracy measures in machine learning we demonstrate the ability to recover true analytic solutions, as opposed to a numerical approximation. The method is verified by assessing its ability to recover known analytic solutions for two separate differential equations. The developed method is compared to a conventional, purely data-driven genetic programmi…
The mean shift (MS) algorithm is a nonparametric method used to cluster sample points and find the local modes of kernel density estimates, using an idea based on iterative gradient ascent. In this paper we develop a mean-shift-inspired algorithm to estimate the modes of regression functions and partition the sample points in the input space. We prove convergence of the sequences generated by the algorithm and derive the non-asymptotic rates of convergence of the estimated local modes for the underlying regression model. We also demonstrate the utility of the algorithm for data-enabled discove…
State Key Lab of Processors, Institute of Computing Technology, CAS（中国科学院计算技术研究所处理器国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Cambricon Technologies（寒武纪科技）
Symbolic regression, the task of extracting mathematical expressions from the observed data $\{ \vx_i, y_i \}$, plays a crucial role in scientific discovery. Despite the promising performance of existing methods, most of them conduct symbolic regression in an \textit{offline} setting. That is, they treat the observed data points as given ones that are simply sampled from uniform distributions without exploring the expressive potential of data. However, for real-world scientific problems, the data used for symbolic regression are usually actively obtained by doing experiments, which is an \text…
In this paper we introduce an active learning method for symbolic regression using StackGP. The approach begins with a small number of data points for StackGP to model. To improve the model the system incrementally adds a data point such that the new point maximizes prediction uncertainty as measured by the model ensemble. Symbolic regression is re-run with the larger data set. This cycle continues until the system satisfies a termination criterion. We use the Feynman AI benchmark set of equations to examine the ability of our method to find appropriate models using fewer data points. The appr…
Massachusetts Institute of Technology（麻省理工学院）；Theiss Research（Theiss研究院）
A core challenge for both physics and artificial intellicence (AI) is symbolic regression: finding a symbolic expression that matches data from an unknown function. Although this problem is likely to be NP-hard in principle, functions of practical interest often exhibit symmetries, separability, compositionality and other simplifying properties. In this spirit, we develop a recursive multidimensional symbolic regression algorithm that combines neural network fitting with a suite of physics-inspired techniques. We apply it to 100 equations from the Feynman Lectures on Physics, and it discovers …
Josef Ressel Centre for Symbolic Regression（约瑟夫·雷塞尔符号回归中心）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）；Johannes Kepler University（约翰内斯·开普勒大学）
We introduce in this paper a runtime-efficient tree hashing algorithm for the identification of isomorphic subtrees, with two important applications in genetic programming for symbolic regression: fast, online calculation of population diversity and algebraic simplification of symbolic expression trees. Based on this hashing approach, we propose a simple diversity-preservation mechanism with promising results on a collection of symbolic regression benchmark problems.
Federal University of ABC（ABC联邦大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）
Symbolic regression is a nonlinear regression method which is commonly performed by an evolutionary computation method such as genetic programming. Quantification of uncertainty of regression models is important for the interpretation of models and for decision making. The linear approximation and so-called likelihood profiles are well-known possibilities for the calculation of confidence and prediction intervals for nonlinear regression models. These simple and effective techniques have been completely ignored so far in the genetic programming literature. In this work we describe the calculat…
Massachusetts Institute of Technology（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Identifying the mathematical relationships that best describe a dataset remains a very challenging problem in machine learning, and is known as Symbolic Regression (SR). In contrast to neural networks which are often treated as black boxes, SR attempts to gain insight into the underlying relationships between the independent variables and the target variable of a given dataset by assembling analytical functions. In this paper, we present GSR, a Generalized Symbolic Regression approach, by modifying the conventional SR optimization problem formulation, while keeping the main SR objective intact…
