DILBERT:用于领域自适应的定制预训练withCategory移位,及其在方面提取中的应用
DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extraction
摘要:
预训练语言模型的兴起在绝大多数自然语言处理(NLP)任务中取得了实质性进展。然而,在某些情况下,针对预训练过程的通用方法自然可能是次优的。特别地,在源域上微调预训练的语言模型,然后将其应用于不同的目标域,导致许多源-目标域对的最终分类器的性能急剧下降。此外,在一些NLP任务中,输出类别在不同领域之间存在很大差异,这使得适应更具挑战性。例如,这发生在方面提取的任务中,其中,餐馆或电子设备可能非常不同。本文提出了一种新的微调方案,BERT,旨在解决上述挑战。我们将此方案命名为DILBERT:领域不变学习与BERT,并定制它的方面提取在无监督领域自适应设置。DILBERT利用源域和目标域的分类信息来指导预训练过程,以实现更领域和类别不变的表示,从而缩小域之间的差距。我们表明,DILBERT产生了实质性的改善,在国家的最先进的基线,同时使用一小部分的未标记的数据,特别是在更具挑战性的领域自适应设置。
英文摘要:
The rise of pre-trained language models has yielded substantial progress in the vast majority of Natural Language Processing (NLP) tasks. However, a generic approach towards the pre-training procedure can naturally be sub-optimal in some cases. Particularly, fine-tuning a pre-trained language model on a source domain and then applying it to a different target domain, results in a sharp performance decline of the eventual classifier for many source-target domain pairs. Moreover, in some NLP tasks, the output categories substantially differ between domains, making adaptation even more challenging. This, for example, happens in the task of aspect extraction, where the aspects of interest of reviews of, e.g., restaurants or electronic devices may be very different. This paper presents a new fine-tuning scheme for BERT, which aims to address the above challenges. We name this scheme DILBERT: Domain Invariant Learning with BERT, and customize it for aspect extraction in the unsupervised domain adaptation setting. DILBERT harnesses the categorical information of both the source and the target domains to guide the pre-training process towards a more domain and category invariant representation, thus closing the gap between the domains. We show that DILBERT yields substantial improvements over state-of-the-art baselines while using a fraction of the unlabeled data, particularly in more challenging domain adaptation setups.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。