VarMAE:用于领域自适应语言理解的变分掩码自编码器预训练
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
摘要:
预训练的语言模型在一般基准测试中取得了令人满意的性能,但在迁移到特定领域时表现不佳。最近的作品执行预训练从零开始或持续预训练领域语料库。然而,在许多特定的领域,有限的语料库很难支持获得精确的表示。为了解决这个问题,我们提出了一种新的基于transformer的语言模型VarMAE域自适应语言理解。在屏蔽自动编码的目标下,我们设计了一个上下文不确定性学习模块,将令牌的上下文编码为平滑的潜在分布。该模块可以产生多样的和良好的形式的上下文表示。在科学和金融领域的NLU任务上的实验表明,VarMAE可以有效地适应资源有限的新领域。
英文摘要:
Pre-trained language models have achieved promising performance on general benchmarks, but underperform when migrated to a specific domain. Recent works perform pre-training from scratch or continual pre-training on domain corpora. However, in many specific domains, the limited corpus can hardly support obtaining precise representations. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。