arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.10907cs.CL
收藏

神经机器翻译中源和目标对预测的贡献分析

Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation

University of Edinburgh(爱丁堡大学);University of Amsterdam(阿姆斯特丹大学);University of Zurich(苏黎世大学)

Elena Voita, Rico Sennrich, Ivan Titov

摘要:

在神经机器翻译(以及更一般的条件语言建模)中,目标标记的生成受到两种类型的上下文的影响:源和目标序列的前缀。虽然许多试图了解NMT模型的内部工作,他们都没有明确评估相对源和目标的贡献,一代的决定。我们认为,这种相对贡献可以通过采用分层相关传播(LRP)的变体进行评估。其潜在的“守恒原理”使相关传播的独特性:不同于其他方法,它评估的不是一个抽象的数量反映令牌的重要性,但每个令牌的影响力的比例。我们将LRP扩展到Transformer,并对NMT模型进行分析,该模型明确评估了源和目标对生成过程的相对贡献。我们分析了这些贡献的变化时,调节不同类型的前缀,当不同的训练目标或训练数据的量,并在训练过程中。我们发现,使用更多数据训练的模型往往更多地依赖于源信息,并且具有更尖锐的令牌贡献;训练过程是非单调的,具有不同性质的几个阶段。

英文摘要:

In Neural Machine Translation (and, more generally, conditional language modeling), the generation of a target token is influenced by two types of context: the source and the prefix of the target sequence. While many attempts to understand the internal workings of NMT models have been made, none of them explicitly evaluates relative source and target contributions to a generation decision. We argue that this relative contribution can be evaluated by adopting a variant of Layerwise Relevance Propagation (LRP). Its underlying 'conservation principle' makes relevance propagation unique: differently from other methods, it evaluates not an abstract quantity reflecting token importance, but the proportion of each token's influence. We extend LRP to the Transformer and conduct an analysis of NMT models which explicitly evaluates the source and target relative contributions to the generation process. We analyze changes in these contributions when conditioning on different types of prefixes, when varying the training objective or the amount of training data, and during the training process. We find that models trained with more data tend to rely on source information more and to have more sharp token contributions; the training process is non-monotonic with several stages of different nature.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑