arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2012.10586cs.CLcs.AI
收藏

领域特定神经机器翻译的稀疏结构发现

Finding Sparse Structures for Domain Specific Neural Machine Translation

ByteDance AI Lab(字节跳动AI实验室)

Jianze Liang, Chengqi Zhao, Mingxuan Wang, Xipeng Qiu, Lei Li

摘要:

神经机器翻译通常采用微调的方法来适应特定的领域。然而,无限制的微调可以很容易地降低一般域和过拟合的目标域。为了缓解这个问题,我们提出了修剪调整,一种新的领域自适应方法,通过逐步修剪。它在对新域进行微调时学习微小的特定于域的子网络。修剪-调整消除了过度拟合和退化问题,而无需修改模型。此外,Prune-Tune能够顺序地学习单个网络,该网络具有用于多个域的多个不相交的特定于域的子网络。实验结果表明,Prune-Tune在目标领域测试集上的性能优于几个强有力的竞争者,而在单领域和多领域测试集上的质量都没有下降。源代码和数据可在https://github.com/ohlionel/Prune-Tune上获得。

英文摘要:

Neural machine translation often adopts the fine-tuning approach to adapt to specific domains. However, nonrestricted fine-tuning can easily degrade on the general domain and over-fit to the target domain. To mitigate the issue, we propose Prune-Tune, a novel domain adaptation method via gradual pruning. It learns tiny domain-specific sub-networks during fine-tuning on new domains. Prune-Tune alleviates the over-fitting and the degradation problem without model modification. Furthermore, Prune-Tune is able to sequentially learn a single network with multiple disjoint domain-specific sub-networks for multiple domains. Empirical experiment results show that Prune-Tune outperforms several strong competitors in the target domain test set without sacrificing the quality on the general domain in both single and multi-domain settings. The source code and data are available at https://github.com/ohlionel/Prune-Tune.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑