一种用于求解算法问题的神经重写系统
A Neural Rewriting System to Solve Algorithmic Problems
AI总结:
针对神经网络难学习算法流程、分布外泛化弱的问题,提出受符号重写系统启发的三模块神经架构,在公式化简任务上优于Neural Data Router和GPT-4,提升了分布外泛化能力。
摘要:
现代神经网络架构在学习算法流程方面仍存在困难,这类流程需要系统地应用组合规则来求解分布外的问题实例。在本研究中,我们聚焦于公式化简问题——这是一类用于研究神经架构系统泛化能力的合成基准。我们提出了一种模块化架构,其设计目标是仅依靠少量训练样本,就能学习求解嵌套数学公式的通用流程。受符号人工智能中的经典框架重写系统启发,我们在该架构中设置了三个相互作用的专用模块:Selector(选择器),经训练用于识别可求解的子表达式;Solver(求解器),负责将子表达式映射为对应的值;Combiner(组合器),用于将原公式中的子表达式替换为求解器给出的解。我们将该系统与Neural Data Router(神经数据路由器,一款近期专为系统泛化设计的模型)、以及采用高级提示策略测试的最先进大语言模型GPT-4进行基准对比。实验证明,在三类不同的公式化简问题上,我们的方法相比这些备选方案实现了更高程度的分布外泛化,同时我们还通过分析其失效情况讨论了该方法的局限性。
英文摘要:
Modern neural network architectures still struggle to learn algorithmic procedures that require to systematically apply compositional rules to solve out-of-distribution problem instances. In this work, we focus on formula simplification problems, a class of synthetic benchmarks used to study the systematic generalization capabilities of neural architectures. We propose a modular architecture designed to learn a general procedure for solving nested mathematical formulas by only relying on a minimal set of training examples. Inspired by rewriting systems, a classic framework in symbolic artificial intelligence, we include in the architecture three specialized and interacting modules: the Selector, trained to identify solvable sub-expressions; the Solver, mapping sub-expressions to their values; and the Combiner, replacing sub-expressions in the original formula with the solution provided by the Solver. We benchmark our system against the Neural Data Router, a recent model specialized for systematic generalization, and a state-of-the-art large language model (GPT-4) probed with advanced prompting strategies. We demonstrate that our approach achieves a higher degree of out-of-distribution generalization compared to these alternative approaches on three different types of formula simplification problems, and we discuss its limitations by analyzing its failures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。