arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.03042cs.CL
收藏

预测-校正增强Transformer与指数移动平均系数学习

Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning

Northeastern University(东北大学);Meituan Inc.(美团公司);Microsoft Research(微软研究院);Tsinghua University(清华大学)

Bei Li, Tong Zheng, Rui Wang, Jiahao Liu, Qingyan Guo, Junliang Guo, Xu Tan, Tong Xiao, Jingbo Zhu, Jingang Wang, Xunliang Cai

AI总结:

本文提出预测-校正增强Transformer,通过高阶预测器与多步校正器最小化ODE截断误差,并采用指数移动平均系数学习,在多项基准上以更少参数超越强基线。

摘要:

残差网络作为常微分方程(ODEs)的离散近似,激发了神经网络设计的重大进展,包括多步方法、高阶方法和多粒子动力系统。ODE解的精度显著影响参数优化,进而影响模型性能。在本工作中,我们提出了一系列Transformer架构设计的高级探索,以最小化与真实“解”相比的误差。首先,我们引入了一个预测-校正学习框架来最小化截断误差,该框架由一个高阶预测器和一个多步校正器组成。其次,我们提出了一种基于指数移动平均的系数学习方法,以增强我们的高阶预测器。在大规模机器翻译、抽象式摘要、语言建模和自然语言理解基准上的广泛实验证明了我们方法的优越性。在WMT'14英德和英法任务上,我们的模型分别取得了30.95和44.27的BLEU分数。此外,在OPUS多语言机器翻译任务上,我们的模型仅使用1/3的参数,平均超过了一个强大的3.8B DeepNet模型2.9个SacreBLEU分数。值得注意的是,在LM Harness评估中,它还比LLama模型高出5.7个准确率点。

英文摘要:

Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep methods, high-order methods, and multi-particle dynamical systems. The precision of the solution to ODEs significantly affects parameter optimization, thereby impacting model performance. In this work, we present a series of advanced explorations of Transformer architecture design to minimize the error compared to the true ``solution.'' First, we introduce a predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and a multistep corrector. Second, we propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor. Extensive experiments on large-scale machine translation, abstractive summarization, language modeling, and natural language understanding benchmarks demonstrate the superiority of our approach. On the WMT'14 English-German and English-French tasks, our model achieved BLEU scores of 30.95 and 44.27, respectively. Furthermore, on the OPUS multilingual machine translation task, our model surpasses a robust 3.8B DeepNet by an average of 2.9 SacreBLEU, using only 1/3 parameters. Notably, it also beats LLama models by 5.7 accuracy points on the LM Harness Evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑