arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.11538cs.CL
收藏

具有鲁棒自训练的跨语言和领域的预训练语言模型的特征适应

Feature Adaptation of Pre-Trained Language Models across Languages and Domains with Robust Self-Training

National University of Singapore(新加坡国立大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Soochow University(苏州大学)

Hai Ye, Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng, Lidong Bing

摘要:

适配预训练的语言模型(PrLM)(例如,BERT)到新领域的研究最近得到了广泛的关注。而不是在大多数以前的工作中所做的微调PrLM,我们研究如何适应新的领域没有微调PrLM的功能。本文研究了无监督领域自适应(UDA)算法。利用PrLM的特征,我们将用标记数据训练的模型从源域调整到未标记的目标域。自训练广泛用于UDA,其预测目标域数据上的伪标签以用于训练。然而,预测的伪标签不可避免地包含噪声,这将对训练鲁棒模型产生负面影响。为了提高自训练的鲁棒性,本文提出了类感知特征自蒸馏(CFd)来从PrLM中学习区分性特征,其中PrLM特征被自蒸馏到特征自适应模块中,并且来自同一类的特征被更紧密地聚类。我们进一步扩展CFd的跨语言设置,在语言差异进行了研究。在两个单语和多语言亚马逊评论数据集上的实验表明,CFd可以持续提高跨领域和跨语言设置中的自我训练性能。

英文摘要:

Adapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has gained much attention recently. Instead of fine-tuning PrLMs as done in most previous work, we investigate how to adapt the features of PrLMs to new domains without fine-tuning. We explore unsupervised domain adaptation (UDA) in this paper. With the features from PrLMs, we adapt the models trained with labeled data from the source domain to the unlabeled target domain. Self-training is widely used for UDA which predicts pseudo labels on the target domain data for training. However, the predicted pseudo labels inevitably include noise, which will negatively affect training a robust model. To improve the robustness of self-training, in this paper we present class-aware feature self-distillation (CFd) to learn discriminative features from PrLMs, in which PrLM features are self-distilled into a feature adaptation module and the features from the same class are more tightly clustered. We further extend CFd to a cross-language setting, in which language discrepancy is studied. Experiments on two monolingual and multilingual Amazon review datasets show that CFd can consistently improve the performance of self-training in cross-domain and cross-language settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑