arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1909.04130cs.CL
收藏

反向迁移学习:针对不同NLP任务训练的词嵌入能否改善神经语言模型?

Reverse Transfer Learning: Can Word Embeddings Trained for Different NLP Tasks Improve Neural Language Models?

KU Leuven(鲁汶大学);Apple Inc.(苹果公司)

Lyan Verwimp, Jerome R. Bellegarda

摘要:

自然语言处理(NLP)任务往往受到缺乏适当注释的训练数据的影响,因此最近迁移学习在各种各样的任务中取得了成功。典型的食谱包括:(i)训练具有与语言建模相关的目标的深度的、可能是双向的神经网络,对于该目标,训练数据是丰富的;以及(ii)使用训练的网络来导出比诸如word 2 vec的标准线性词嵌入丰富得多的上下文表示,并且因此导致重要的增益。在这项工作中,我们想知道相反的观点是否也是正确的:为不同的NLP任务训练的上下文表示是否可以改善语言建模本身?由于语言模型(LM)主要是局部优化的,其他NLP任务可以帮助它们基于文档的整个语义结构做出更好的预测。我们测试了神经LM中几种类型的预训练嵌入的性能,并研究了是否有可能通过用域分类模型预训练的嵌入使LM更了解全局语义信息。初步实验表明,只要在训练过程中使用适当的客观标准,预训练的嵌入可能对神经语言建模有益。

英文摘要:

Natural language processing (NLP) tasks tend to suffer from a paucity of suitably annotated training data, hence the recent success of transfer learning across a wide variety of them. The typical recipe involves: (i) training a deep, possibly bidirectional, neural network with an objective related to language modeling, for which training data is plentiful; and (ii) using the trained network to derive contextual representations that are far richer than standard linear word embeddings such as word2vec, and thus result in important gains. In this work, we wonder whether the opposite perspective is also true: can contextual representations trained for different NLP tasks improve language modeling itself? Since language models (LMs) are predominantly locally optimized, other NLP tasks may help them make better predictions based on the entire semantic fabric of a document. We test the performance of several types of pre-trained embeddings in neural LMs, and we investigate whether it is possible to make the LM more aware of global semantic information through embeddings pre-trained with a domain classification model. Initial experiments suggest that as long as the proper objective criterion is used during training, pre-trained embeddings are likely to be beneficial for neural language modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑