我们提出了一种用于符号回归（SR）的新方法，该任务旨在寻找能够最佳解释数据集的紧凑程序化假设。该问题通常通过遗传算法解决；我们表明，可以通过归纳一个抽象文本概念库来增强此类方法。我们的算法名为 LaSR，使用对大型语言模型（LLM）的零样本查询来发现并演化出现在已知高性能假设中的概念。我们结合标准演化步骤和以已发现概念为条件、由 LLM 引导的步骤（通过零样本 LLM 查询获得）来发现新假设。假设一旦被发现，就会用于新一轮的概念抽象和演化。我们在常用 SR 基准 Feynman 方程以及一组合成任务上验证了 LaSR。在这些基准上，LaSR 显著优于多种基于深度学习和演化算法的最先进 SR 方法。此外，我们表明 LaSR 可用于发现一条新颖且强大的 LLM 缩放定律。
英文摘要：
We present a novel method for symbolic regression (SR), the task of searching for compact programmatic hypotheses that best explain a dataset. The problem is commonly solved using genetic algorithms; we show that we can enhance such methods by inducing a library of abstract textual concepts. Our algorithm, called LaSR, uses zero-shot queries to a large language model (LLM) to discover and evolve concepts occurring in known high-performing hypotheses. We discover new hypotheses using a mix of standard evolutionary steps and LLM-guided steps (obtained through zero-shot LLM queries) conditioned on discovered concepts. Once discovered, hypotheses are used in a new round of concept abstraction and evolution. We validate LaSR on the Feynman equations, a popular SR benchmark, as well as a set of synthetic tasks. On these benchmarks, LaSR substantially outperforms a variety of state-of-the-art SR approaches based on deep learning and evolutionary algorithms. Moreover, we show that LaSR can be used to discover a novel and powerful scaling law for LLMs.
AutoVSR: Automatic Visual-to-Symbolic Reasoning for Symbolic Expression Generation from Circuit Schematic
Hunan University, Changsha, China（湖南大学）；Wuhan University of Technology, Wuhan, China（武汉理工大学）；Huazhong University of Science and Technology, Wuhan, China（华中科技大学）
Michigan State University（密歇根州立大学）；Sichuan University（四川大学）；Florida Institute for Human and Machine Cognition（佛罗里达人类与机器认知研究所）
We demonstrate a library for the integration of domain knowledge in deep learning architectures. Using this library, the structure of the data is expressed symbolically via graph declarations and the logical constraints over outputs or latent variables can be seamlessly added to the deep models. The domain knowledge can be defined explicitly, which improves the models' explainability in addition to the performance and generalizability in the low-data regime. Several approaches for such an integration of symbolic and sub-symbolic models have been introduced; however, there is no library to faci…
LieDiscover: Adaptive Symbolic Library Construction for Explicit Open-form Symmetry Discovery
East China Normal University（华东师范大学）；Shanghai Jiao Tong University（上海交通大学）；Beihang University（北京航空航天大学）；Institute of Applied Physics and Computational Mathematics（应用物理与计算数学研究所）；Northeast Normal University（东北师范大学）
Mathematical reasoning recently has been shown as a hard challenge for neural systems. Abilities including expression translation, logical reasoning, and mathematics knowledge acquiring appear to be essential to overcome the challenge. This paper demonstrates that some abilities can be achieved through abductive combination with discrete systems that have been programmed with human knowledge. On a mathematical reasoning dataset, we adopt the recently proposed abductive learning framework, and propose the ABL-Sym algorithm that combines the Transformer neural models with a symbolic mathematics …
Width-based planning methods have been shown to yield state-of-the-art performance in the Atari 2600 domain using pixel input. One successful approach, RolloutIW, represents states with the B-PROST boolean feature set. An augmented version of RolloutIW, $π$-IW, shows that learned features can be competitive with handcrafted ones for width-based search. In this paper, we leverage variational autoencoders (VAEs) to learn features directly from pixels in a principled manner, and without supervision. The inference model of the trained VAEs extracts boolean features from pixels, and RolloutIW plans…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Rice University（莱斯大学）
Recent advances in TCP congestion control (CC) have achieved tremendous success with deep reinforcement learning (RL) approaches, which use feedforward neural networks (NN) to learn complex environment conditions and make better decisions. However, such "black-box" policies lack interpretability and reliability, and often, they need to operate outside the traditional TCP datapath due to the use of complex NNs. This paper proposes a novel two-stage solution to achieve the best of both worlds: first to train a deep RL agent, then distill its (over-)parameterized NN policy into white-box, light-w…
MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
U. Washington（华盛顿大学）；CMU（卡内基梅隆大学）；Stanford（斯坦福大学）；NUS（新加坡国立大学）；TuSimple（图森未来）；NYU（纽约大学）；Microsoft（微软）；U. Alberta（阿尔伯塔大学）；MIT（麻省理工学院）；NYU Shanghai（上海纽约大学）
School of Computing and Augmented Intelligence（计算与增强智能学院）；Arizona State University（亚利桑那州立大学）
Reinforcement learning in problems with symbolic state spaces is challenging due to the need for reasoning over long horizons. This paper presents a new approach that utilizes relational abstractions in conjunction with deep learning to learn a generalizable Q-function for such problems. The learned Q-function can be efficiently transferred to related problems that have different object names and object quantities, and thus, entirely different state spaces. We show that the learned generalized Q-function can be utilized for zero-shot transfer to related problems without an explicit, hand-coded…
Dual embodied-symbolic concept representations are the foundation for deep learning and symbolic AI integration. We discuss the use of dual embodied-symbolic concept representations for molecular graph representation learning, specifically with exemplar-based contrastive self-supervised learning (SSL). The embodied representations are learned from molecular graphs, and the symbolic representations are learned from the corresponding Chemical knowledge graph (KG). We use the Chemical KG to enhance molecular graphs with symbolic (semantic) knowledge and generate their augmented molecular graphs. …
Discovery of Nonlinear Dynamics with Automated Basis Function Generation
Data Science and Analytics Institute, University of Oklahoma, Norman, OK, USA（数据科学与分析研究所，俄克拉荷马大学，诺曼，OK，USA）；School of Industrial and Systems Engineering, University of Oklahoma, Norman, OK, USA（工业与系统工程学院，俄克拉荷马大学，诺曼，OK，USA）
The symbolic AI community is increasingly trying to embrace machine learning in neuro-symbolic architectures, yet is still struggling due to cultural barriers. To break the barrier, this rather opinionated personal memo attempts to explain and rectify the conventions in Statistics, Machine Learning, and Deep Learning from the viewpoint of outsiders. It provides a step-by-step protocol for designing a machine learning system that satisfies a minimum theoretical guarantee necessary for being taken seriously by the symbolic AI community, i.e., it discusses "in what condition we can stop worrying …
Imperial College London（帝国理工学院）；University of Bamberg（班贝格大学）
Given the recent successes of Deep Learning in AI there has been increased interest in the role and need for explanations in machine learned theories. A distinct notion in this context is that of Michie's definition of Ultra-Strong Machine Learning (USML). USML is demonstrated by a measurable increase in human performance of a task following provision to the human of a symbolic machine learned theory for task performance. A recent paper demonstrates the beneficial effect of a machine learned logic theory for a classification task, yet no existing work to our knowledge has examined the potentia…
MIT-IBM Watson AI Lab, IBM Research（麻省理工学院-IBM沃森人工智能实验室，IBM研究院）
We achieved a new milestone in the difficult task of enabling agents to learn about their environment autonomously. Our neuro-symbolic architecture is trained end-to-end to produce a succinct and effective discrete state transition model from images alone. Our target representation (the Planning Domain Definition Language) is already in a form that off-the-shelf solvers can consume, and opens the door to the rich array of modern heuristic search capabilities. We demonstrate how the sophisticated innate prior we place on the learning process significantly reduces the complexity of the learned r…
University of Trento（特伦托大学）；University of Pisa（比萨大学）
Neuro-symbolic predictors learn a mapping from sub-symbolic inputs to higher-level concepts and then carry out (probabilistic) logical inference on this intermediate representation. This setup offers clear advantages in terms of consistency to symbolic prior knowledge, and is often believed to provide interpretability benefits in that - by virtue of complying with the knowledge - the learned concepts can be better understood by human stakeholders. However, it was recently shown that this setup is affected by reasoning shortcuts whereby predictions attain high accuracy by leveraging concepts wi…
APPCAIR, BITS Pilani K.K. Birla Goa Campus（BITS Pilani K.K. Birla Goa校区APPCAIR）；TCS Research（塔塔咨询服务研究部）；IIT Delhi（印度理工学院德里分校）
We model short-duration (e.g. day) trading in financial markets as a sequential decision-making problem under uncertainty, with the added complication of continual concept-drift. We, therefore, employ meta reinforcement learning via the RL2 algorithm. It is also known that human traders often rely on frequently occurring symbolic patterns in price series. We employ logical program induction to discover symbolic patterns that occur frequently as well as recently, and explore whether using such features improves the performance of our meta reinforcement learning algorithm. We report experiments …
From Hybrid Mechanistic--Data-Driven Modeling Toward Neuro-Symbolic AI: What, Why, and How
Institute for Computational Biomedicine, RWTH Aachen University（计算生物医学研究所，亚琛工业大学）；Center for Computational Life Sciences, RWTH Aachen University（计算生命科学中心，亚琛工业大学）
Recently deep reinforcement learning has achieved tremendous success in wide ranges of applications. However, it notoriously lacks data-efficiency and interpretability. Data-efficiency is important as interacting with the environment is expensive. Further, interpretability can increase the transparency of the black-box-style deep RL models and hence gain trust from the users. In this work, we propose a new hierarchical framework via symbolic RL, leveraging a symbolic transition model to improve the data-efficiency and introduce the interpretability for learned policy. This framework consists o…
Stanford University（斯坦福大学）；California Institute of Technology（加州理工学院）
Humans have the remarkable ability to recognize and acquire novel visual concepts in a zero-shot manner. Given a high-level, symbolic description of a novel concept in terms of previously learned visual concepts and their relations, humans can recognize novel concepts without seeing any examples. Moreover, they can acquire new concepts by parsing and communicating symbolic structures using learned visual concepts and relations. Endowing these capabilities in machines is pivotal in improving their generalization capability at inference time. In this work, we introduce Zero-shot Concept Recognit…
Princeton University（普林斯顿大学）；DeepMind（深度思维）；New York University（纽约大学）；Flatiron Institute（弗莱提荣研究所）；Carnegie Mellon University（卡内基梅隆大学）
We develop a general approach to distill symbolic representations of a learned deep model by introducing strong inductive biases. We focus on Graph Neural Networks (GNNs). The technique works as follows: we first encourage sparse latent representations when we train a GNN in a supervised setting, then we apply symbolic regression to components of the learned model to extract explicit physical relations. We find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark…
Cornell University（康奈尔大学）；Brookhaven National Laboratory（布鲁克海文国家实验室）；Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室）
We employ variational autoencoders to extract physical insight from a dataset of one-particle Anderson impurity model spectral functions. Autoencoders are trained to find a low-dimensional, latent space representation that faithfully characterizes each element of the training set, as measured by a reconstruction error. Variational autoencoders, a probabilistic generalization of standard autoencoders, further condition the learned latent space to promote highly interpretable features. In our study, we find that the learned latent variables strongly correlate with well known, but nontrivial, par…
Symbolic Recovery of Differential Equations: The Identifiability Problem
Ludwig-Maximilians-Universität München（慕尼黑路德维希-马克西米利安大学）；Technische Universität München（慕尼黑技术大学）；Munich Center for Quantum Science and Technology (MCQST)（慕尼黑量子科学与技术中心）；Munich Quantum Valley (MQV)（慕尼黑量子谷）；University of Tromsø（特罗姆瑟大学）；DLR-German Aerospace Center（德国航空航天中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
