ODE Transformer：一种基于常微分方程的序列生成模型
ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation
摘要：
残差网络是常微分方程（ODE）解的欧拉离散化。本文探讨了Transformer和数值常微分方程方法之间更深层次的关系。我们首先表明，在Transformer层的残留块可以被描述为一个高阶的解决方案，常微分方程。受此启发，我们设计了一个新的体系结构，{ \it ODE Transformer}，这是类似的龙格库塔方法，这是很好的动机在ODE。作为对Transformer的自然扩展，ODE Transformer易于实现，使用效率高。在大规模机器翻译、摘要摘要和语法纠错任务上的实验结果表明，ODE Transformer具有很高的通用性。它可以在强基线上获得模型性能的大幅改善（例如，30.77和44.11 BLEU分数的WMT'14英语-德语和英语-法语基准）在推理效率略有损失。
英文摘要：
Residual networks are an Euler discretization of solutions to Ordinary Differential Equations (ODE). This paper explores a deeper relationship between Transformer and numerical ODE methods. We first show that a residual block of layers in Transformer can be described as a higher-order solution to ODE. Inspired by this, we design a new architecture, {\it ODE Transformer}, which is analogous to the Runge-Kutta method that is well motivated in ODE. As a natural extension to Transformer, ODE Transformer is easy to implement and efficient to use. Experimental results on the large-scale machine translation, abstractive summarization, and grammar error correction tasks demonstrate the high genericity of ODE Transformer. It can gain large improvements in model performance over strong baselines (e.g., 30.77 and 44.11 BLEU scores on the WMT'14 English-German and English-French benchmarks) at a slight cost in inference efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
