arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.10439cs.CLcs.LG
收藏

ViDeBERTa:一个强大的越南语预训练语言模型

ViDeBERTa: A powerful pre-trained language model for Vietnamese

FPT AI Center(FPT人工智能中心);Microsoft(微软);University of California San Diego(加州大学圣地亚哥分校);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Cong Dao Tran, Nhut Huy Pham, Anh Nguyen, Truong Son Hy, Tu Vu

摘要:

本文介绍了ViDeBERTa,一个新的预训练的单语语言模型越南语,有三个版本- ViDeBERTa_xsmall,ViDeBERTa_base和ViDeBERTa_large,这是预先训练的大规模语料库的高质量和多样化的越南语文本使用DeBERTa架构。尽管许多基于Transformer的成功的预训练语言模型已经被广泛地提出用于英语语言,但是仍然很少有用于越南语的预训练模型,越南语是一种低资源语言,在下游任务上表现良好,特别是问答。我们对三个重要的自然语言下游任务,词性标注,命名实体识别和问答,我们的模型进行微调和评估。实证结果表明,ViDeBERTa与少得多的参数超过了以前的国家的最先进的模型在多个越南特定的自然语言理解任务。值得注意的是,具有86 M参数的ViDeBERTa_base仅为具有370 M参数的PhoBERT_large的约23%,仍然比以前的最先进模型执行相同或更好的结果。我们的ViDeBERTa模型可在https://github.com/HySonLab/ViDeBERTa上获得。

英文摘要:

This paper presents ViDeBERTa, a new pre-trained monolingual language model for Vietnamese, with three versions - ViDeBERTa_xsmall, ViDeBERTa_base, and ViDeBERTa_large, which are pre-trained on a large-scale corpus of high-quality and diverse Vietnamese texts using DeBERTa architecture. Although many successful pre-trained language models based on Transformer have been widely proposed for the English language, there are still few pre-trained models for Vietnamese, a low-resource language, that perform good results on downstream tasks, especially Question answering. We fine-tune and evaluate our model on three important natural language downstream tasks, Part-of-speech tagging, Named-entity recognition, and Question answering. The empirical results demonstrate that ViDeBERTa with far fewer parameters surpasses the previous state-of-the-art models on multiple Vietnamese-specific natural language understanding tasks. Notably, ViDeBERTa_base with 86M parameters, which is only about 23% of PhoBERT_large with 370M parameters, still performs the same or better results than the previous state-of-the-art model. Our ViDeBERTa models are available at: https://github.com/HySonLab/ViDeBERTa.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑