arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.10315cs.CL
收藏

桥接预训练的语言模型和手工制作的功能,用于无监督的POS标记

Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging

Soochow University(苏州大学)

Houquan Zhou, Yang Li, Zhenghua Li, Min Zhang

摘要:

近年来,大规模预训练的语言模型(PLM)在大多数NLP任务中取得了非凡的进展。但是,在无监督的词性标注任务中,利用PLM的作品很少,并且无法达到最先进的(SOTA)性能。最近的SOTA性能是由He等人提出的Guassian HMM变体产生的。(2018)。然而,作为一种生成模型,HMM做出了非常强的独立性假设,这使得从PLM中整合上下文化的单词表示非常具有挑战性。在这项工作中,我们第一次提出了一个神经条件随机场自动编码器(CRF-AE)模型的无监督词性标注。CRF-AE的鉴别编码器可以直接合并ELMo单词表示。此外,灵感来自功能丰富的HMM,我们重新引入手工制作的功能到CRF-AE的解码器。最后,实验清楚地表明,我们的模型在Penn Treebank和多语言Universal Advertisements Treebank v2.0上的性能大大优于以前的最先进模型。

英文摘要:

In recent years, large-scale pre-trained language models (PLMs) have made extraordinary progress in most NLP tasks. But, in the unsupervised POS tagging task, works utilizing PLMs are few and fail to achieve state-of-the-art (SOTA) performance. The recent SOTA performance is yielded by a Guassian HMM variant proposed by He et al. (2018). However, as a generative model, HMM makes very strong independence assumptions, making it very challenging to incorporate contexualized word representations from PLMs. In this work, we for the first time propose a neural conditional random field autoencoder (CRF-AE) model for unsupervised POS tagging. The discriminative encoder of CRF-AE can straightforwardly incorporate ELMo word representations. Moreover, inspired by feature-rich HMM, we reintroduce hand-crafted features into the decoder of CRF-AE. Finally, experiments clearly show that our model outperforms previous state-of-the-art models by a large margin on Penn Treebank and multilingual Universal Dependencies treebank v2.0.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑