PDKEEPAAA:一种简单有效的无成本改进Transformer的技术
UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
摘要:
Transformer体系结构在丰富的自然语言处理任务中取得了巨大的成功。Transformer模型的过参数化已经激发了大量的工作来减轻其过拟合以获得更好的性能。通过一些探索,我们发现简单的技术,如dropout,可以大大提高模型的性能与精心设计。因此,在本文中,我们将不同的dropout技术集成到Transformer模型的训练中。具体来说,我们提出了一种名为UniDrop的方法,将从细粒度到粗粒度的三种不同的dropout技术结合起来,即,特征丢失、结构丢失和数据丢失。理论上,我们证明了这三个辍学发挥不同的作用,从正规化的角度来看。从经验上讲,我们在神经机器翻译和文本分类基准数据集上进行了实验。广泛的结果表明,Transformer with UniDrop可以在IWSLT14翻译任务上实现约1.5 BLEU的改进,并且即使使用强大的预训练RoBERTA作为骨干,也可以获得更好的分类准确性。
英文摘要:
Transformer architecture achieves great success in abundant natural language processing tasks. The over-parameterization of the Transformer model has motivated plenty of works to alleviate its overfitting for superior performances. With some explorations, we find simple techniques such as dropout, can greatly boost model performance with a careful design. Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unites three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play different roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classification benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for the classification even using strong pre-trained RoBERTa as backbone.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。