arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2004.02199cs.CLcs.LG
收藏

理解神经机器翻译的学习动力学

Understanding Learning Dynamics for Neural Machine Translation

Harbin Institute of Technology(哈尔滨工业大学);Tencent AI Lab(腾讯人工智能实验室)

Conghui Zhu, Guanlin Li, Lemao Liu, Tiejun Zhao, Shuming Shi

摘要:

尽管NMT取得了巨大的成功,但仍然存在一个严峻的挑战:很难解释其训练过程中的内部动态。在本文中,我们建议理解学习动力学的NMT使用最近提出的技术命名为损失变化分配(LCA)~ \citep{lan-2019-loss-change-allocation}。由于LCA需要为每次更新计算整个数据集的梯度,因此我们提出了一个近似值,以便在NMT场景中将其付诸实践。%的人从新加坡发展局的教训中获得动力。我们的模拟实验表明,这样的近似计算是有效的,并根据经验证明,提供一致的结果,蛮力实现。特别是,在两个标准翻译基准数据集上的广泛实验揭示了一些有价值的发现。

英文摘要:

Despite the great success of NMT, there still remains a severe challenge: it is hard to interpret the internal dynamics during its training process. In this paper we propose to understand learning dynamics of NMT by using a recent proposed technique named Loss Change Allocation (LCA)~\citep{lan-2019-loss-change-allocation}. As LCA requires calculating the gradient on an entire dataset for each update, we instead present an approximate to put it into practice in NMT scenario. %motivated by the lesson from sgd. Our simulated experiment shows that such approximate calculation is efficient and is empirically proved to deliver consistent results to the brute-force implementation. In particular, extensive experiments on two standard translation benchmark datasets reveal some valuable findings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑