arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.00430cs.CLcs.AI
收藏

VarMAE:用于领域自适应语言理解的变分掩码自编码器预训练

VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding

Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Ping An Life Insurance Company of China, Ltd.(中国平安人寿保险股份有限公司)

Dou Hu, Xiaolong Hou, Xiyang Du, Mengyuan Zhou, Lianxin Jiang, Yang Mo, Xiaofeng Shi

摘要:

预训练的语言模型在一般基准测试中取得了令人满意的性能,但在迁移到特定领域时表现不佳。最近的作品执行预训练从零开始或持续预训练领域语料库。然而,在许多特定的领域,有限的语料库很难支持获得精确的表示。为了解决这个问题,我们提出了一种新的基于transformer的语言模型VarMAE域自适应语言理解。在屏蔽自动编码的目标下,我们设计了一个上下文不确定性学习模块,将令牌的上下文编码为平滑的潜在分布。该模块可以产生多样的和良好的形式的上下文表示。在科学和金融领域的NLU任务上的实验表明,VarMAE可以有效地适应资源有限的新领域。

英文摘要:

Pre-trained language models have achieved promising performance on general benchmarks, but underperform when migrated to a specific domain. Recent works perform pre-training from scratch or continual pre-training on domain corpora. However, in many specific domains, the limited corpus can hardly support obtaining precise representations. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑