ODE Transformer：一个基于常微分方程的神经机器翻译模型
ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation
摘要：
已经发现，残差网络是常微分方程（ODE）的解的欧拉离散化。在本文中，我们探讨了Transformer和常微分方程数值方法之间更深层次的关系。我们表明，在Transformer层的残留块可以被描述为一个高阶的解决方案常微分方程。这导致我们设计一个新的架构（称之为ODE Transformer）类似的龙格库塔方法，在常微分方程的动机。作为对Transformer的自然扩展，ODE Transformer具有易于实现和参数高效的特点.我们在三个WMT任务上的实验证明了该模型的通用性，以及在几个强基线上的性能大幅提高。在WMT'14 En-De和En-Fr测试数据上，其BLEU得分分别为30.76和44.11。这为WMT'14 En-Fr任务设定了一个新的最先进水平。
英文摘要：
It has been found that residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODEs). In this paper, we explore a deeper relationship between Transformer and numerical methods of ODEs. We show that a residual block of layers in Transformer can be described as a higher-order solution to ODEs. This leads us to design a new architecture (call it ODE Transformer) analogous to the Runge-Kutta method that is well motivated in ODEs. As a natural extension to Transformer, ODE Transformer is easy to implement and parameter efficient. Our experiments on three WMT tasks demonstrate the genericity of this model, and large improvements in performance over several strong baselines. It achieves 30.76 and 44.11 BLEU scores on the WMT'14 En-De and En-Fr test data. This sets a new state-of-the-art on the WMT'14 En-Fr task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
