Neural sequence models trained with maximum likelihood estimation have led to breakthroughs in many tasks, where success is defined by the gap between training and test performance. However, their ability to achieve stronger forms of generalization remains unclear. We consider the problem of symbolic mathematical integration, as it requires generalizing systematically beyond the test set. We develop a methodology for evaluating generalization that takes advantage of the problem domain's structure and access to a verifier. Despite promising in-distribution performance of sequence-to-sequence models in this domain, we demonstrate challenges in achieving robustness, compositionality, and out-of-distribution generalization, through both carefully constructed manual test suites and a genetic algorithm that automatically finds large collections of failures in a controllable manner. Our investigation highlights the difficulty of generalizing well with the predominant modeling and learning approach, and the importance of evaluating beyond the test set, across different aspects of generalization.
Adaptive Problem Generation via Symbolic Representations
Korea Advanced Institute of Science and Technology(韩国科学技术院);Singapore-MIT Alliance for Research and Technology(新加坡-麻省理工联盟研究技术);Massachusetts Institute of Technology(麻省理工学院);Singapore Management University(新加坡管理大学)
A Neural-Guided Dynamic Symbolic Network for Exploring Mathematical Expressions from Data
AnnLab(安澜实验室);Institute of Semiconductors, Chinese Academy of Sciences(中国科学院半导体研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Key Laboratory of Semiconductor Neural Network Intelligent Sensing and Computing Technology(北京市半导体神经网络智能感知与计算重点实验室);Center of Materials Science and Optoelectronics Engineering, University of Chinese Academy of Sciences(中国科学院大学材料科学与光电技术学院)