记忆注意融合:基于转换器的Sequence-to-Sequence模型的外部语言模型集成
Memory Attentive Fusion: External Language Model Integration for Transformer-based Sequence-to-Sequence Model
摘要:
本文提出了一种新的融合方法,将外部语言模型(LM)集成到基于Transformer的序列到序列(seq 2seq)模型中。虽然配对数据基本上需要训练seq 2seq模型,但外部LM可以仅使用未配对数据进行训练。因此,重要的是利用外部LM中存储的知识来构建seq 2seq模型,因为很难准备大量的配对数据。然而,现有的融合方法假设LM与基于递归神经网络的seq 2seq模型而不是Transformer集成。因此,本文提出了一种融合方法,可以显式地利用网络结构中的Transformer。所提出的方法,称为{ \bf记忆注意力融合},利用变压器式的注意力机制,重复源-目标的注意力,在多跳的方式读取记忆的知识LM。实验结果表明,该方法优于传统的融合方法。
英文摘要:
This paper presents a novel fusion method for integrating an external language model (LM) into the Transformer based sequence-to-sequence (seq2seq) model. While paired data are basically required to train the seq2seq model, the external LM can be trained with only unpaired data. Thus, it is important to leverage memorized knowledge in the external LM for building the seq2seq model, since it is hard to prepare a large amount of paired data. However, the existing fusion methods assume that the LM is integrated with recurrent neural network-based seq2seq models instead of the Transformer. Therefore, this paper proposes a fusion method that can explicitly utilize network structures in the Transformer. The proposed method, called {\bf memory attentive fusion}, leverages the Transformer-style attention mechanism that repeats source-target attention in a multi-hop manner for reading the memorized knowledge in the LM. Our experiments on two text-style conversion tasks demonstrate that the proposed method performs better than conventional fusion methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。