Friction systems are mechanical systems wherein friction is used for force transmission (e.g. mechanical braking systems or automatic gearboxes). For finding optimal and safe design parameters, engineers have to predict friction system performance. This is especially difficult in real-world applications, because it is affected by many parameters. We have used symbolic regression and genetic programming for finding accurate and trustworthy prediction models for this task. However, it is not straight-forward how nominal variables can be included. In particular, a one-hot-encoding is unsatisfactory because genetic programming tends to remove such indicator variables. We have therefore used so-called factor variables for representing nominal variables in symbolic regression models. Our results show that GP is able to produce symbolic regression models for predicting friction performance with predictive accuracy that is comparable to artificial neural networks. The symbolic regression models with factor variables are less complex than models using a one-hot encoding.
Josef Ressel Centre for Symbolic Regression（约瑟夫·雷塞尔符号回归中心）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）；Johannes Kepler University（约翰内斯·开普勒大学）
We introduce in this paper a runtime-efficient tree hashing algorithm for the identification of isomorphic subtrees, with two important applications in genetic programming for symbolic regression: fast, online calculation of population diversity and algebraic simplification of symbolic expression trees. Based on this hashing approach, we propose a simple diversity-preservation mechanism with promising results on a collection of symbolic regression benchmark problems.
Discovering Continuous-Time Memory-Based Symbolic Policies using Genetic Programming
Department of Machine Learning and Neural Computing（机器学习与神经计算系）；Donders Institute for Brain, Cognition and Behaviour（大脑、认知与行为研究所）；Radboud University（拉德堡德大学）
Heuristic and Evolutionary Algorithms Laboratory（启发式与进化算法实验室）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）；Institute for Formal Models and Verification（形式模型与验证研究所）；Johannes Kepler University Linz（林茨约翰开普勒大学）
This paper describes a methodology for analyzing the evolutionary dynamics of genetic programming (GP) using genealogical information, diversity measures and information about the fitness variation from parent to offspring. We introduce a new subtree tracing approach for identifying the origins of genes in the structure of individuals, and we show that only a small fraction of ancestor individuals are responsible for the evolvement of the best solutions in the population.
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Symbolic regression is the process of identifying mathematical expressions that fit observed output from a black-box process. It is a discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic programming. In this work, we introduce a hybrid neural-guided/genetic programming approach to symbolic regression and other combinatorial optimization problems. We propose a neural-guided component used to seed the starting population of a random restart genetic programming component…
Evaluation of Population Initialization Methods for Genetic Programming-based Symbolic Regression
Heuristic and Evolutionary Algorithms Laboratory, University of Applied Sciences Upper Austria（上奥地利应用科学大学启发式与进化算法实验室）；Astrophysics, University of Oxford（牛津大学天体物理学系）；Institute of Cosmology & Gravitation, University of Portsmouth（朴茨茅斯大学宇宙学与引力研究所）
University of Utah（犹他大学）；Scientific Computing and Imaging Institute（科学计算与成像研究所）；NASA Langley Research Center（美国国家航空航天局兰利研究中心）
In this paper, we present a machine learning method for the discovery of analytic solutions to differential equations. The method utilizes an inherently interpretable algorithm, genetic programming based symbolic regression. Unlike conventional accuracy measures in machine learning we demonstrate the ability to recover true analytic solutions, as opposed to a numerical approximation. The method is verified by assessing its ability to recover known analytic solutions for two separate differential equations. The developed method is compared to a conventional, purely data-driven genetic programmi…
Peng Cheng Laboratory（鹏城实验室）；University of Washington（华盛顿大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
Partial differential equations (PDEs) are concise and understandable representations of domain knowledge, which are essential for deepening our understanding of physical processes and predicting future responses. However, the PDEs of many real-world problems are uncertain, which calls for PDE discovery. We propose the symbolic genetic algorithm (SGA-PDE) to discover open-form PDEs directly from data without prior knowledge about the equation structure. SGA-PDE focuses on the representation and optimization of PDE. Firstly, SGA-PDE uses symbolic mathematics to realize the flexible representatio…
Federal University of ABC（ABC联邦大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）
Symbolic regression is a nonlinear regression method which is commonly performed by an evolutionary computation method such as genetic programming. Quantification of uncertainty of regression models is important for the interpretation of models and for decision making. The linear approximation and so-called likelihood profiles are well-known possibilities for the calculation of confidence and prediction intervals for nonlinear regression models. These simple and effective techniques have been completely ignored so far in the genetic programming literature. In this work we describe the calculat…
Symbolic Regression searches for a function form that approximates a dataset often using Genetic Programming. Since there is usually no restriction to what form the function can have, Genetic Programming may return a hard to understand model due to non-linear function chaining or long expressions. A novel representation called Interaction-Transformation was recently proposed to alleviate this problem. In this representation, the function form is restricted to an affine combination of terms generated as the application of a single univariate function to the interaction of selected variables. Th…
Interpretable Robotic Friction Learning via Symbolic Regression
Ludwig-Maximilians-Universität München（慕尼黑路易斯-马克西米利安大学）；German Aerospace Center（德国航空航天中心）；University of Tromsø（特罗姆瑟大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
The Interaction-Transformation (IT) is a new representation for Symbolic Regression that restricts the search space into simpler, but expressive, function forms. This representation has the advantage of creating a smoother search space unlike the space generated by Expression Trees, the common representation used in Genetic Programming. This paper introduces an Evolutionary Algorithm capable of evolving a population of IT expressions supported only by the mutation operator. The results show that this representation is capable of finding better approximations to real-world data sets when compar…
The Zoetrope Genetic Programming (ZGP) algorithm is based on an original representation for mathematical expressions, targeting evolutionary symbolic regression.The zoetropic representation uses repeated fusion operations between partial expressions, starting from the terminal set. Repeated fusions within an individual gradually generate more complex expressions, ending up in what can be viewed as new features. These features are then linearly combined to best fit the training data. ZGP individuals then undergo specific crossover and mutation operators, and selection takes place between parent…
Adaptive Protection for Evolutionary Feature Construction in Symbolic Regression with Application to Credit Classification
Centre for Data Science and Artificial Intelligence（数据科学与人工智能中心）；School of Engineering and Computer Science（工程与计算机科学学院）；Victoria University of Wellington（惠灵顿维多利亚大学）；Business School, Sichuan University（四川大学商学院）；Michigan State University（密歇根州立大学）
In this paper we provide a broad benchmarking of recent genetic programming approaches to symbolic regression in the context of state of the art machine learning approaches. We use a set of nearly 100 regression benchmark problems culled from open source repositories across the web. We conduct a rigorous benchmarking of four recent symbolic regression approaches as well as nine machine learning approaches from scikit-learn. The results suggest that symbolic regression performs strongly compared to state-of-the-art gradient boosting algorithms, although in terms of running times is among the sl…
Neuro-Evolutionary Approach to Physics-Aware Symbolic Regression
Czech Institute of Informatics, Robotics, and Cybernetics CTU in Prague（捷克信息学、机器人学和自动化研究中心CTU布拉格分校）；Dept. of Cognitive Robotics, TU Delft（认知机器人学系代尔夫特理工大学）；CIIRC, CTU in Prague（智能信息研究中心CTU布拉格分校）
University of Applied Sciences Upper Austria（上奥地利应用科学大学）；MCL Forschungs GmbH（MCL研究有限责任公司）
We describe an approach for empirical modeling of steel phase kinetics based on symbolic regression and genetic programming. The algorithm takes processed data gathered from dilatometer measurements and produces a system of differential equations that models the phase kinetics. Our initial results demonstrate that the proposed approach allows to identify compact differential equations that fit the data. The model predicts ferrite, pearlite and bainite formation for a single steel type. Martensite is not yet included in the model. Future work shall incorporate martensite and generalize to multi…
Efficient Constant Optimization for Symbolic Regression with GPU-Accelerated Tree-Based Genetic Programming
The Hong Kong Polytechnic University（香港理工大学）；University of Washington（华盛顿大学）；Amazon Web Services（亚马逊网络服务）；Jiangxi University of Finance and Economics（江西财经大学）；EEO Education Technology（EEO教育科技）；Chongqing Medical University（重庆医科大学）；Eastern Institute of Technology（东方理工学院）
Université de Paris（巴黎大学）；Ochanomizu University（御茶水女子大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
By combining Genetic Programming, MAP-Elites and Covariance Matrix Adaptation Evolution Strategy, we demonstrate very high success rates in Symbolic Regression problems. MAP-Elites is used to improve exploration while preserving diversity and avoiding premature convergence and bloat. Then, a Covariance Matrix Adaptation-Evolution Strategy is used to evaluate free scalars through a non-gradient-based black-box optimizer. Although this evaluation approach is not computationally scalable to high dimensional problems, our algorithm is able to find exactly most of the $31$ targets extracted from th…
Josef Ressel Center for Symbolic Regression（约瑟夫·雷塞尔符号回归中心）；Heuristic and Evolutionary Algorithms Laboratory（启发式与进化算法实验室）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）
The growing volume of data makes the use of computationally intense machine learning techniques such as symbolic regression with genetic programming more and more impractical. This work discusses methods to reduce the training data and thereby also the runtime of genetic programming. The data is aggregated in a preprocessing step before running the actual machine learning algorithm. K-means clustering and data binning is used for data aggregation and compared with random sampling as the simplest data reduction method. We analyze the achieved speed-up in training and the effects on the trained …
Enhancing Generalization in Evolutionary Feature Construction for Symbolic Regression through Vicinal Jensen Gap Minimization
Centre for Data Science and Artificial Intelligence & School of Engineering and Computer Science, Victoria University of Wellington（数据科学与人工智能中心及工程与计算机科学学院，维多利亚大学）
University of Applied Sciences Upper Austria（上奥地利应用科技大学）；Johannes Kepler University（约翰内斯·开普勒大学）
Symbolic regression is a powerful system identification technique in industrial scenarios where no prior knowledge on model structure is available. Such scenarios often require specific model properties such as interpretability, robustness, trustworthiness and plausibility, that are not easily achievable using standard approaches like genetic programming for symbolic regression. In this chapter we introduce a deterministic symbolic regression algorithm specifically designed to address these issues. The algorithm uses a context-free grammar to produce models that are parameterized by a non-line…
Biomimetic Machine Learning approach for prediction of mechanical properties of Additive Friction Stir Deposited Aluminum alloys based walled structures
Meta AI；Sorbonne Université（索邦大学）；Ecole Normale Supérieure（巴黎高等师范学院）
Symbolic regression, the task of predicting the mathematical expression of a function from the observation of its values, is a difficult task which usually involves a two-step procedure: predicting the "skeleton" of the expression up to the choice of numerical constants, then fitting the constants by optimizing a non-convex loss function. The dominant approach is genetic programming, which evolves candidates by iterating this subroutine a large number of times. Neural networks have recently been tasked to predict the correct skeleton in a single try, but remain much less powerful. In this pape…
Tecnológico Nacional de México/IT de Tijuana（墨西哥国家理工学院/蒂胡ana信息技术学院）；LASIGE, Department of Informatics, Faculty of Sciences, University of Lisbon（里斯本大学科学学院信息系LASIGE）
Symbolic regression is the task of identifying a mathematical expression that best fits a provided dataset of input and output values. Due to the richness of the space of mathematical expressions, symbolic regression is generally a challenging problem. While conventional approaches based on genetic evolution algorithms have been used for decades, deep learning-based methods are relatively new and an active research area. In this work, we present SymbolicGPT, a novel transformer-based language model for symbolic regression. This model exploits the advantages of probabilistic language models lik…
China University of Petroleum-Beijing（中国石油大学（北京））；University of Wisconsin Milwaukee（威斯康星大学密尔沃基分校）
Many recent studies focus on developing mechanisms to explain the black-box behaviors of neural networks (NNs). However, little work has been done to extract the potential hidden semantics (mathematical representation) of a neural network. A succinct and explicit mathematical representation of a NN model could improve the understanding and interpretation of its behaviors. To address this need, we propose a novel symbolic regression method for neural works (called SRNet) to discover the mathematical expressions of a NN. SRNet creates a Cartesian genetic programming (NNCGP) to represent the hidd…
Leiden University Medical Center（莱顿大学医学中心）；Centrum Wiskunde & Informatica（荷兰国家数学与计算机科学研究所）
Dimensionality reduction (DR) is an important technique for data exploration and knowledge discovery. However, most of the main DR methods are either linear (e.g., PCA), do not provide an explicit mapping between the original data and its lower-dimensional representation (e.g., MDS, t-SNE, isomap), or produce mappings that cannot be easily interpreted (e.g., kernel PCA, neural-based autoencoder). Recently, genetic programming (GP) has been used to evolve interpretable DR mappings in the form of symbolic expressions. There exists a number of ways in which GP can be used to this end and no study…
From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification
Faculty of Engineering & Information Technology, University of Technology Sydney（工程与信息技术学院，技术悉尼大学）；Department of Civil and Architectural Engineering, Sultan Qaboos University（土木与建筑学院，苏丹·卡布斯大学）；University Research and Innovation Center (EKIK), Obuda University（研究与创新中心（EKIK），布达佩斯大学）
Toward Physically Plausible Data-Driven Models: A Novel Neural Network Approach to Symbolic Regression
Czech Technical University in Prague（布拉格捷克理工大学）；Delft University of Technology（代尔夫特理工大学）；Czech Institute of Informatics, Robotics, and Cybernetics（捷克信息学、机器人学与控制论研究所）
Josef Ressel Centre for Symbolic Regression（约瑟夫·雷塞尔符号回归中心）；Heuristic and Evolutionary Algorithms Laboratory（启发式与进化算法实验室）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）
We describe a method for the identification of models for dynamical systems from observational data. The method is based on the concept of symbolic regression and uses genetic programming to evolve a system of ordinary differential equations (ODE). The novelty is that we add a step of gradient-based optimization of the ODE parameters. For this we calculate the sensitivities of the solution to the initial value problem (IVP) using automatic differentiation. The proposed approach is tested on a set of 19 problem instances taken from the literature which includes datasets from simulated systems a…
National Aeronautics and Space Administration（美国国家航空航天局）；University of Utah（犹他大学）
Interpretability and uncertainty quantification in machine learning can provide justification for decisions, promote scientific discovery and lead to a better understanding of model behavior. Symbolic regression provides inherently interpretable machine learning, but relatively little work has focused on the use of symbolic regression on noisy data and the accompanying necessity to quantify uncertainty. A new Bayesian framework for genetic-programming-based symbolic regression (GPSR) is introduced that uses model evidence (i.e., marginal likelihood) to formulate replacement probability during …
Recently, several algorithms for symbolic regression (SR) emerged which employ a form of multiple linear regression (LR) to produce generalized linear models. The use of LR allows the algorithms to create models with relatively small error right from the beginning of the search; such algorithms are thus claimed to be (sometimes by orders of magnitude) faster than SR algorithms based on vanilla genetic programming. However, a systematic comparison of these algorithms on a common set of problems is still missing. In this paper we conceptually and experimentally compare several representatives of…
Massachusetts Institute of Technology（麻省理工学院）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Identifying the mathematical relationships that best describe a dataset remains a very challenging problem in machine learning, and is known as Symbolic Regression (SR). In contrast to neural networks which are often treated as black boxes, SR attempts to gain insight into the underlying relationships between the independent variables and the target variable of a given dataset by assembling analytical functions. In this paper, we present GSR, a Generalized Symbolic Regression approach, by modifying the conventional SR optimization problem formulation, while keeping the main SR objective intact…
Symbolic-Diffusion: Deep Learning Based Symbolic Regression with D3PM Discrete Token Diffusion
Department of Mechatronics Engineering University of Waterloo（机械工程系大学水力学院）；Department of Computer Science University of Waterloo（计算机科学系大学水力学院）；Department of Statistics and Actuarial Science University of Waterloo（统计与精算科学系大学水力学院）
Centro de investigación CITIC（CITIC研究中心）；University of A Coruña（拉科鲁尼亚大学）
This paper describes a new method for Symbolic Regression that allows to find mathematical expressions from a dataset. This method has a strong mathematical basis. As opposed to other methods such as Genetic Programming, this method is deterministic, and does not involve the creation of a population of initial solutions. Instead of it, a simple expression is being grown until it fits the data. The experiments performed show that the results are as good as other Machine Learning methods, in a very low computational time. Another advantage of this technique is that the complexity of the expressi…
University of Applied Sciences Upper Austria（上奥地利应用科技大学）；Johannes Kepler University（约翰内斯·开普勒大学）
In this chapter we take a closer look at the distribution of symbolic regression models generated by genetic programming in the search space. The motivation for this work is to improve the search for well-fitting symbolic regression models by using information about the similarity of models that can be precomputed independently from the target function. For our analysis, we use a restricted grammar for uni-variate symbolic regression models and generate all possible models up to a fixed length limit. We identify unique models and cluster them based on phenotypic as well as genotypic similarity…
SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compression
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Massachusetts Institute of Technology（麻省理工学院）；Institute of Physics Belgrade（贝尔格莱德物理研究所）；Institute for Artificial Intelligence and Fundamental Interactions（人工智能与基本相互作用研究所）
