增强符号常微分方程表达式的Transformer表示
Enhancing Transformer Representations of Symbolic ODE Expressions
- University of Southampton（南安普顿大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究探索树位置嵌入与对比学习增强Transformer对符号ODE表达式的结构化表示，提升学习性能并支持数学交换性质。
AI中文摘要：
现有的求解微分方程的方法，如符号回归、物理信息神经网络和神经算子，通常侧重于数值近似或通过拟合数值数据进行盲目的符号搜索。对于学习数学表达式的结构化表示（这些表示保留交换性质并能支持符号形式的数学推理）的关注较少。Transformer模型在求解符号微分方程方面已展现出强大的能力。然而，标准的位置嵌入是为序列数据设计的。符号微分方程天然由表达式树表示，因此这些位置嵌入可能无法有效捕获其层次结构。我们研究了符号常微分方程（ODE）任务中现有的树位置嵌入。我们系统地研究了它们在不同设置下的有效性。结果表明，树位置嵌入有助于早期训练轮次的学习，并在整个训练过程中持续提升性能，最终在各种数据规模和任务上带来一致的改进。基于学习到的结构化表示，我们应用对比学习来支持数学中的交换性质。消融研究深入揭示了这些方法在建模符号数学结构时如何相互作用。
英文摘要：
Existing approaches to solving differential equations, such as symbolic regression, physics informed neural networks, and neural operators, typically focus on numerical approximations or blind symbolic search via fitting to numerical data. Less attention has been paid to learning structured representations of mathematical expressions that preserve commutative properties and could support mathematical reasoning in symbolic forms. Transformer models have shown strong capabilities in solving symbolic differential equations. However, standard positional embeddings in transformers are designed for sequence data. Symbolic differential equations are naturally represented by expression trees, so these positional embeddings may not efficiently capture their hierarchical structures. We investigate existing tree positional embeddings in symbolic ordinary differential equation (ODE) tasks. We systematically study their effectiveness under different settings. Our results show that tree positional embeddings aid learning in early epochs and continue to improve performance throughout, ultimately yielding consistent advantages across various data sizes and tasks. Based on learned structural representations, we apply contrastive learning to support the commutative property in mathematics. Ablation studies provide insight into how these methods interact in modelling symbolic mathematical structures.
